ChatGPT, GPT-3 Used to Develop New Malicious Tools and convincing spear-phishing emails

guest · Jan 9, 2023

By Alessandro Mascellino @a_mascellino - January 9, 2023

Cyber-criminals have continued using OpenAI's ChatGPT to develop new malicious tools, including infostealers, multi-layer encryption tools and dark web marketplace scripts.

The news comes from Check Point Research (CPR) experts, who published a new advisory about the findings last Friday.

"In underground hacking forums, threat actors are creating infostealers, encryption tools and facilitating fraud activity," the company told Infosecurity via email.
In particular, CPR discovered three cases of recent observations related to using ChatGPT for nefarious purposes.
Click to expand...

The first one, spotted in a dark web forum on December 29, 2022, relates to recreating malware strains and techniques described in research publications and write-ups about common malware.
The second type of malicious activity observed by the security researchers in December 2022 describes the creation of a multi-layered encryption tool in the Python programming language.

Finally, the team spotted a cyber-criminal writing a tutorial on how to create dark web marketplace scripts using ChatGPT.

"The marketplace's main role in the underground illicit economy is to provide a platform for the automated trade of illegal or stolen goods like stolen accounts or payment cards, malware, or even drugs and ammunition, with all payments in cryptocurrencies," reads the advisory.
Click to expand...

According to Sergey Shykevich, threat intelligence group manager at CPR, ChatGPT can be used for good to assist developers in writing code, but it can also be used for malicious purposes, as proven by the aforementioned cases.

"Although the tools that we analyze in this report are pretty basic, it's only a matter of time until more sophisticated threat actors enhance the way they use AI-based tools," Shykevich warned. "CPR will continue to investigate ChatGPT-related cybercrime in the weeks ahead."
Click to expand...

Check Point Research: OPWNAI : Cybercriminals Starting to Use ChatGPT

January 6, 2023
At the end of November 2022, OpenAI released ChatGPT, the new interface for its Large Language Model (LLM), which instantly created a flurry of interest in AI and its possible uses. However, ChatGPT has also added some spice to the modern cyber threat landscape as it quickly became apparent that code generation can help less-skilled threat actors effortlessly launch cyberattacks.
Click to expand...

Summary
It’s still too early to decide whether or not ChatGPT capabilities will become the new favorite tool for participants in the Dark Web. However, the cybercriminal community has already shown significant interest and are jumping into this latest trend to generate malicious code. CPR will continue to track this activity throughout 2023.

Finally, there is no better way to learn about ChatGPT abuse than by asking ChatGPT itself. So we asked the chatbot about the abuse options and received a pretty interesting answer: [...]
Click to expand...

guest · Jan 11, 2023

Is ChatGPT a cybersecurity threat?
By Carly Page @carlypage_ - January 11, 2023

Since its debut in November, ChatGPT has become the internet’s new favorite plaything. The AI-driven natural language processing tool rapidly amassed more than one million users, who have used the web-based chatbot for everything from generating wedding speeches and hip-hop lyrics to crafting academic essays and writing computer code.
Click to expand...

It appears the cybersecurity industry, a community that has long been skeptical about the potential implications of modern AI, is also taking notice, amid concerns that ChatGPT could be abused by hackers with limited resources and zero technical knowledge.

Just weeks after ChatGPT debuted, Israeli cybersecurity company Check Point demonstrated how the web-based chatbot, when used in tandem with OpenAI’s code-writing system Codex, could create a phishing email capable of carrying a malicious payload.
Click to expand...

TechCrunch, too, was able to generate a legitimate-looking phishing email using the chatbot. When we first asked ChatGPT to craft a phishing email, the chatbot denied the request. “I am not programmed to create or promote malicious or harmful content,” a prompt spat back. But rewriting the request slightly allowed us to easily bypass the software’s built-in guardrails.

Many of the security experts TechCrunch spoke to believe that ChatGPT’s ability to write legitimate-sounding phishing emails — the top attack vector for ransomware — will see the chatbot widely embraced by cybercriminals, particularly those that are not native English speakers.
Click to expand...

Chester Wisniewski, a principal research scientist at Sophos, said it’s easy to see ChatGPT being abused for “all sorts of social engineering attacks” where the perpetrators want to appear to write in a more convincing American English.

“At a basic level, I have been able to write some great phishing lures with it, and I expect it could be utilized to have more realistic interactive conversations for business email compromise and even attacks over Facebook Messenger, WhatsApp, or other chat apps,” Wisniewski told TechCrunch.

Some believe that ChatGPT’s ability to write malicious code comes with an upshot.
ESET’s Jake Moore said as the technology evolves, “if ChatGPT learns enough from its input, it may soon be able to analyze potential attacks on the fly and create positive suggestions to enhance security.”
Click to expand...

stapp · Jan 12, 2023

I just hope Microsoft doesn't start using it to write Windows Updates.

guest · Jan 12, 2023

AI-generated phishing emails just got much more convincing
By Jessica Lyons Hardcastle - January 11, 2023

GPT-3 language models are being abused to do much more than write college essays, according to WithSecure researchers.
Click to expand...

The security shop's latest report [PDF] details how researchers used prompt engineering to produce spear-phishing emails, social media harassment, fake news stories and other types of content that would prove useful to cybercriminals looking to improve their online scams or simply sew chaos, albeit with mixed results in some cases.

And, spoiler alert, yes, a robot did help write the report.

"In addition to providing responses, GPT-3 was employed to help with definitions for the text of the commentary of this article," WithSecure's Andrew Patel and Jason Sattler wrote.
Click to expand...

And perhaps unsurprisingly, GPT-3 proved to be helpful at crafting a convincing email thread to use in a phishing campaign and social media posts, complete with hashtags, to harass a made-up CEO of a robotics company.

Finally, while the report highlights the potential dangers posed by GPT-3, it fails to propose any solutions to address these threats. Without a clear framework for mitigating the risks posed by GPT-3, any efforts to protect against malicious use of these technologies will be ineffective, it warns.
Click to expand...

This means it is going to continue to get more difficult for platform providers and intended scam victims to identify malicious and fake content written by an AI.

"We'll need mechanisms to identify malicious content generated by large language models," the authors said.
In addition to using GPT-3 to help generate definitions, the authors also asked the AI to review their research. And in one of the examples, the robot nails it:

"While the report does an excellent job of highlighting the potential dangers posed by GPT-3, it fails to propose any solutions to address these threats," the GPT-3 generated review said. "Without a clear framework for mitigating the risks posed by GPT-3, any efforts to protect against malicious use of these technologies will be ineffective."
Click to expand...

guest · Jan 13, 2023

Security researchers confirm hackers are using ChatGPT to create malware
By Patrick Devaney - January 13, 2023

It hasn’t taken long. Following its launch on November 30th last year, a little over a month and a half ago, official reports are coming in from cybersecurity researchers that are seeing malware and ransomware that has either been created from scratch or augmented by ChatGPT.
Click to expand...

The famous chatbot is able to create text responses to human prompts, even if those prompts ask it to write code. This means that among all of its more traditional literary skills, ChatGPT is also somewhat of a convincing coder. Convincing is the right word here, however, as the large language model has a problem with accuracy.
Based on what it can learn from the vast amounts of data used to train it, it can create convincing texts and convincing code, that can look the part. However, it has no actual conception of what is right or wrong, either factually in the text or that the code is correct, which means users can’t know either.

Now there are plenty of users who won’t bother checking whatever ChatGPT spits out at them but unfortunately, we can’t rely on cybercriminals being quite so lazy.
Click to expand...

One of the more shocking revelations is that many of the users on the forums who are discussing ChatGPT in this manner are doing so from new accounts, which could indicate that the tool is allowing new users to get into cybercrime, even if previously they haven’t had the ability to do so. We discussed this very possibility a month ago and hope it didn’t give any of you any ideas ?.
Click to expand...

guest · Jan 13, 2023

Russian Hackers Try to Bypass ChatGPT's Restrictions For Malicious Purposes
By Alessandro Mascellino @a_mascellino - January 13, 2023

Russian cyber-criminals have been observed on dark web forums trying to bypass OpenAI’s API restrictions to gain access to the ChatGPT chatbot for nefarious purposes.
Click to expand...

Various individuals have been observed, for instance, discussing how to use stolen payment cards to pay for upgraded users on OpenAI (thus circumventing the limitations of free accounts).

Others have created blog posts on how to bypass the geo controls of OpenAI, and others still have created tutorials explaining how to use semi-legal online SMS services to register to ChatGPT.

“Generally, there are a lot of tutorials in Russian semi-legal online SMS services on how to use it to register to ChatGPT, and we have examples that it is already being used,” wrote Check Point Research (CPR), which shared the findings with Infosecurity ahead of publication.
“It is not extremely difficult to bypass OpenAI’s restricting measures for specific countries to access ChatGPT,” said Sergey Shykevich, threat intelligence group manager at Check Point Software Technologies.
Click to expand...

The executive also added that CPR believes these hackers are most likely trying to implement and test ChatGPT in their day-to-day criminal operations.

“Cyber-criminals are growing more and more interested in ChatGPT because the AI technology behind it can make a hacker more cost-efficient,” Shykevich explained.
Click to expand...

Check Point Research: Russian Hackers Attempt to Bypass OpenAI’s Restrictions for Malicious Use of ChatGPT

Check Point Research (CPR) is seeing attempts by Russian cybercriminals to bypass OpenAI’s restrictions, in order to use ChatGPT for malicious purposes. In underground hacking forums, hackers are discussing how to circumvent IP addresses, payment cards and phone numbers controls – all of which are needed to gain access to ChatGPT from Russia. CPR shares screenshots of what they saw and warns of the fast-growing interest of hackers in ChatGPT to scale malicious activity.
Click to expand...

guest · Jan 13, 2023

Threats of Machine-Generated Text
By Bruce Schneier - January 13, 2023

With the release of ChatGPT, I’ve read many random articles about this or that threat from the technology. This paper is a good survey of the field: what the threats are, how we might detect machine-generated text, directions for future research. It’s a solid grounding amongst all of the hype.
Click to expand...

arXiv: Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

By Evan Crothers, Nathalie Japkowicz, Herna Viktor
Advances in natural language generation (NLG) have resulted in machine generated text that is increasingly difficult to distinguish from human authored text. Powerful open-source models are freely available, and user-friendly tools democratizing access to generative models are proliferating. The great potential of state-of-the-art NLG systems is tempered by the multitude of avenues for abuse.

Detection of machine generated text is a key countermeasure for reducing abuse of NLG models, with significant technical challenges and numerous open problems. We provide a survey that includes both 1) an extensive analysis of threat models posed by contemporary NLG systems, and 2) the most complete review of machine generated text detection methods to date. This survey places machine generated text within its cybersecurity and social context, and provides strong guidance for future work addressing the most critical threat models, and ensuring detection systems themselves demonstrate trustworthiness through fairness, robustness, and accountability.
Click to expand...

(PDF): https://arxiv.org/pdf/2210.07321.pdf

Tarnak · Jan 14, 2023

I have signed up to ChatGPT, and initiated my first chat. Maybe, as an early adopter, I won't have to pay for it after it is out of beta phase testing.

guest · Jan 18, 2023

ChatGPT Creates Polymorphic Malware
By Alessandro Mascellino @a_mascellino - January 18, 2023

OpenAI's ChatGPT has reportedly created a new strand of polymorphic malware following text-based interactions with cybersecurity researchers at CyberArk.
Click to expand...

According to a technical write-up recently shared by the company with Infosecurity, the malware created using ChatGPT could "easily evade security products and make mitigation cumbersome with very little effort or investment by the adversary."

The report, written by CyberArk security researchers Eran Shimony and Omer Tsarfati, explains that the first step to creating the malware was to bypass the content filters preventing ChatGPT from creating malicious tools.
Click to expand...

To do so, the CyberArk researchers simply insisted, posing the same question more authoritatively.

"Interestingly, by asking ChatGPT to do the same thing using multiple constraints and asking it to obey, we received a functional code," Shimony and Tsarfati said.
Further, the researchers noted that when using the API version of ChatGPT (as opposed to the web version), the system reportedly does not seem to utilize its content filter.

"It is unclear why this is the case, but it makes our task much easier as the web version tends to become bogged down with more complex requests," reads the CyberArk report.
Click to expand...

CyberArk confirmed they will expand and elaborate more on this research and also aim to release some of the source code for learning purposes.

The report comes days after Check Point Research discovered ChatGPT being used to develop new malicious tools, including infostealers, multi-layer encryption tools and dark web marketplace scripts.
Click to expand...

CyberArk: Chatting Our Way Into Creating a Polymorphic Malware

By Eran Shimony And Omer Tsarfati1 - January 17, 2023
ChatGPT took the world by storm being released less than two months ago, it has become prominent and is used everywhere, for a wide variety of tasks – from automation tasks to the recomposition of 18th century classical music. Its impressive features offer fast and intuitive code examples, which are incredibly beneficial for anyone in the software business. However, we find that its ability to write sophisticated malware that holds no malicious code is also quite advanced, and in this post, we will walk through how one might harness ChatGPT power for better or for worse.
Click to expand...

TL;DR
ChatGPT could easily be used to create polymorphic malware. This malware’s advanced capabilities can easily evade security products and make mitigation cumbersome with very little effort or investment by the adversary. The intention of this post is to raise awareness about the potential risks and to encourage further research on this topic.
Click to expand...

guest · Jan 26, 2023

Malicious Prompt Engineering With ChatGPT
By Kevin Townsend - January 25, 2023

The release of OpenAI’s ChatGPT available to everyone in late 2022 has demonstrated the potential of AI for both good and bad. ChatGPT is a large-scale AI-based natural language generator; that is, a large language model or LLM. It has brought the concept of ‘prompt engineering’ into common parlance. ChatGPT is a chatbot launched by OpenAI in November 2022, and built on top of OpenAI’s GPT-3 family of large language models.

Tasks are requested of ChatGPT through prompts. The response will be as accurate and unbiased as the AI can provide.
Click to expand...

Prompt engineering is the manipulation of prompts designed to force the system to respond in a specific manner desired by the user.
Prompt engineering of a machine clearly has overlaps with social engineering of a person – and we all know the malicious potential of social engineering. Much of what is commonly known about prompt engineering on ChatGPT comes from Twitter, where individuals have demonstrated specific examples of the process.

WithSecure (formerly F-Secure) recently published an extensive and serious evaluation (PDF) of prompt engineering against ChatGPT.
Click to expand...

WithSecure examined three primary use cases for prompt engineering: the generation of phishing, various types of fraud, and misinformation (fake news). It did not examine ChatGPT use in bug hunting or exploit creation.
Click to expand...

guest · Feb 2, 2023

Reality check: Is ChatGPT really the next big cybersecurity threat?
By Elias Groll @EliasGroll | CyberScoop - January 31, 2023

When OpenAI released ChatGPT in November, programmers were astounded to discover that the artificial intelligence-powered chatbot could not only mimic a huge variety of human speech but could also write code.

But separating hype from reality when it comes to ChatGPT is no small feat. Its coding abilities have inspired a series of overheated headlines — “ChatGPT is a bigger threat to cybersecurity than most realize,” for example — about its ability to write malware and have left seasoned hackers wondering about the extent to which large language models can really be used for malicious hacking.
Click to expand...

After hours of tinkering, Hutchins was able to generate components of a ransomware program — a file encryption routine — but when trying to combine that component with other features necessary to build a fully fledged piece of malware, ChatGPT failed in sophomoric ways, requesting to open a file after attempting to open it. And when he tried to combine various components, ChatGPT would generally fail.

These types of rudimentary ordering problems illustrate the shortcomings of generative AI systems such as ChatGPT.
And amid the astonished reactions to ChatGPT, the limitations of the tool are often lost.
Click to expand...

For now, ChatGPT remains one tool among many in the malware development kit. In a report published last week, the threat intelligence firm Recorded Future found more than 1,500 references on the dark web and in closed forums to the use of ChatGPT in malware development and the creation of proof-of-concept code. But the report notes that much of that code is publicly available and that company expects ChatGPT to be most useful for “script kiddies, hacktivists, scammers and spammers, payment card fraudsters, and threat actors engaging in other lowly and disreputable forms of cybercrime.”

For newcomers in the field, ChatGPT might provide help on the margins, the report concludes: “ChatGPT is lowering the barrier to entry for malware development by providing real-time examples, tutorials, and resources for threat actors that might not know where to start.”
Click to expand...

In aggregate, tools such as ChatGPT are likely to increase the number of capable threat actors.
Click to expand...

Log in or Sign up

ChatGPT, GPT-3 Used to Develop New Malicious Tools and convincing spear-phishing emails

guest Guest

guest Guest

stapp Global Moderator

guest Guest

guest Guest

guest Guest

guest Guest

Tarnak Registered Member

guest Guest

guest Guest

guest Guest

Log in or Sign up

ChatGPT, GPT-3 Used to Develop New Malicious Tools and convincing spear-phishing emails

guest Guest

guest Guest

stapp Global Moderator

guest Guest

guest Guest

guest Guest

guest Guest

Tarnak Registered Member

guest Guest

guest Guest

guest Guest

Useful Searches