Problem/Motivation

In the past, Drupal served "Site builders", and in the quickly changing world, Drupal can adapt to serve "AI builders".

Drupal has great "Views" module. A flexible system enabling users to create various of things. Drupal could have similar building blocks for simplified creation of AI solutions as well (by integrating and combining various APIs like Azure Cognitive Services, OpenAI and others)

Proposed resolution

This issue is to track all integration efforts across Drupal ecosystem to research best ways to address complicated AI building problems. Ongoing research results and related issues will posted in the comments.

 

Drupal artificial intelligence (AI) community initiative

Let's make great Drupal AI features
CommentFileSizeAuthor
#3 chatGPT and Azure Voice.png50.64 KBmindaugasd
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mindaugasd created an issue. See original summary.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Overview of Drupal AI modules in one place

Constantly updated.

Grouped by category:

AI integration within Drupal interface

AI tools

AI building

Indexing Drupal content for AI context

Integrations of easy to use commercial packages

  • Quickchat (quickchat). Quickchat AI empowers companies to build their own multilingual AI assistants powered by generative AI models such as GPT‑3.
  • AI Sidekick (sidekick). AI Sidekick is focused on helping editors to create content faster, brainstorm new ideas, and turn thoughts into brilliant text.
  • Chatbase (chatbase). AI-powered chatbots that streamline customer interaction. It provides sophisticated automation, customization capabilities, 24/7 customer support, and insights for business growth.
  • Leap AI (leap_ai). Integration of tryleap.ai service for image generation
  • Txt42 - ChatGPT copywriting plugin for CKEditor 4 (txt42). Integration of txt42.ai service for writing text

Other:

Have not reviewed yet:

Empty modules:

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented
FileSize
chatGPT and Azure Voice.png50.64 KB

Which module will be first to implement ChatGPT API to make CHAT using drupal, and to make it more capable than original ChatGPT?

For example:

  1. combining it with Azure Text-To-Speech to make it speak?
  2. or combining it with OpenAI's Image generation to make it draw pictures?
  3. or combining it with some Web Search API to provide up-to-date answers (like Bing Chat)?

Sending a sketch I drawed using ECA module displaying how a speaking CHAT might look like.

Sketch drawed with ECA modeller

  • It is a high level sketch.
  • Condition "Can update?" really means "Does a user have enough account balance (and other conditions) to make this request". In other words, each item represents more complex functionality behind it.
  • Actual functionality (lower level building parts) don't exist yet. And it is not clear yet how to make them.
mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Drupal AI news - 2023 March 10

The best modules in different categories

Module first to implement CHAT with ChatGPT API

No one yet.

Recent key articles in Drupal AI world

Related issues:

Communication channels:

  • Come join central slack channel #ai
  • There are other channels (and more active) for different modules which can be found on module project pages.
mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

I asked GPT-4 to generate user story examples for development of Drupal AI. I encourage everyone to chat with AI about Drupal.

Also explored creating an AI module using GPT-4 itself. Published case study here: #3336313-6: Use ChatGPT for solving Drupal issues to increase rate of development

1. As a content creator, I want the AI to generate article drafts based on given keywords or topics, so that I can save time in the content creation process and focus on refining the generated drafts.

2. As a website owner, I want to provide personalized content recommendations to my users based on their browsing history and preferences, so that I can enhance user engagement and increase the time users spend on my website.

3. As an e-commerce website administrator, I want an AI-powered chatbot that can intelligently guide users through the product catalog and answer frequently asked questions, so that users can find the information they need quickly and the support team can focus on more complex queries.

4. As a site builder, I want a collection of pre-built AI-powered components, such as blocks and widgets, that I can easily drag and drop into the site design, so that I can add AI functionality to the site without extensive coding.

5. As a multilingual website owner, I want to use AI to automatically translate my site's content into multiple languages, so that I can reach a broader audience and cater to users with diverse language preferences.

6. As a community manager, I want an AI-powered content moderation system that automatically removes any spam or offensive content from my site, so that the online environment remains safe and welcoming for all users.

7. As a marketing manager, I want AI-driven sentiment analysis of user comments and reviews on my website, so I can better understand user opinions and preferences and adjust our marketing strategies accordingly.

8. As a developer, I want access to a well-documented AI Framework API, so I can build custom AI-powered features and extend the core functionality of the Drupal AI module easily.

murrayw’s picture

murrayw CreditAttribution: murrayw at Morpht commented

Thanks for the review of the modules. I am responding as one of the maintainers of Augmentor. It is nice to see that it is recognised as being flexible and with good code quality. I'd like to provide some more background on Augmentor.

Firstly, the module was created out of a recognition that many AI solutions in Druapl (2019 to 2022) were bespoke and tended to solve the same issues over and over. eg. The development of widgets in the node edit form to expose the AI functionality. So Augmentor aimed to provide an implementation for these UI features so that future AI modules did not need to reinvent the wheel on this.

Secondly, the module was created to provide a pluggable system for AI integrations. A review was done of the common patterns for how AI modules worked (config, input, keyed array of outputs) - the concept of an "Augmentor" was based on this. So there is a key abstraction of what an AI integration would look like.

Finally, we built out an ecosystem of important integrations: Open AI (GPT3 completion), Google Text to Speech, Google Speech to Text, Google Cloud Vision. This kicked off the ecosystem. Recently we have added OpenAI ChatGPT and NLP Cloud. Adding a new integration is pretty straight forward as all the developer needs to handle is the connection and the config for how the integration is to work.

We are encouraging the development of a vibrant ecosystem around the module. It would be great to see the addition of new sub modules supporting new AI systems. We are also working on extending the intergration points into Drupal. If you are considering working on AI in Drupal, I would encourage you to take a look at the Augmentor AI module and its ecosystem.
https://www.drupal.org/project/augmentor/ecosystem

murrayw’s picture

murrayw CreditAttribution: murrayw at Morpht commented

It is also worth adding some background to Quickchat AI module, as there are some relevant insights from that project which will be relevant to future AI integrations in Drupal.

Qucikchat AI is a paid service which prevides chatbot functionality on top of OpenAI. One fo the interesting things about Quickchat is that it provides an API for syncing across a "knowledge base" to Quickchat Ai which is used in defining the custom content for the chatbot. ie. The content in Drupal, is driving the experience. When a user submits a question to the chatbot, the knowledge base can be referred to before getting the response from OpenAI.

This kind of model - Drupal as the content source, will no doubt be a pattern which is repeated in the future as deeper AI integrations are made which use the CMS as the source of the content.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Thanks for creating Augmentor ecosystem. Here found a nice video about it https://www.youtube.com/watch?v=iwxD7rMmnCQ

Interesting point:

Drupal as the content source, will no doubt be a pattern which is repeated in the future as deeper AI integrations are made which use the CMS as the source of the content.

There is an issue of DrupalGPT #3349218: Training a Large Language Model for Code (Code LLM) - an expert Drupal A.I. to create a robot with very high degree of Drupal expertise, which could help develop Drupal itself.

But also making an AI aware and use content of any Drupal website in an interesting angle.
Beyond Drupal, everyone across the world will need this functionality to train A.I. in all kinds of knowledge ecosystems. I wonder what the accessible and thorough solution to this will be.

murrayw’s picture

murrayw CreditAttribution: murrayw at Morpht commented

Back again with a few more thoughts about building a chat bot and extending on from the knowledge base comments for Qucikchat AI...

It is important to note that LLMs come pretrained. It is is possible to finetune and prompt engineer to get certain results. However, in order to get the most from a knowledge base it is required to use embeddings to understand the content and what is being asked of it. The basic ideas is that

  • Your knowledge base is analysed up front for semantic meaning.
  • Then the question asked by the user undergoes a similar process so that the most relevant pieces of information can be extracted from the knowledge base.
  • Once these parts have been found, they are sumbitted to the LLM with a prompt to get a conversational response.

See the following helpful articles on how this can be achieved:

So, in a nutshell, the content in Drupal will need to be analysed first, so that the chatbot can be customised to the content on the site in question. Quickchat AI makes this process easier because it handles the embeddings, the prompt and finally the widget (with voice integration) to make it all work. The Quickchat AI module is really just about submitting the knowledge base to Quickchat which then does the processing work. ie. Drupal is used as the source of truth for the content and the functionality and processing is offloaded to an external service.

Of course, this functionality could be implemented in Drupal, The above tutorials show how that could be done. It would however require a an intermediate step before ECA came into the picture. Building and maintaining a database of embeddings for site content is a potential way forward for someone wanting to support this funtionality.

The future - it is easy to see that OpenAI (and other AI providers) will start injesting sites automatically and create embeddings for them. Search engines will not just be for keywords, but for semantic emeddings of meaning. The chatbot functionality could then be easily added to the site. This will probably happen very quickly and so, the effort to manage this in Drupal may be wasted.

From my point of view, using Drupal to maintain a knowledge base which can be used as a source for services such as Quickchat AI is a sensible way forward as it leverages Drupal for what it is good at.

Finally, this doesn't leave ECA out of the picture. Processing nodes and other entities via ECA and Augmentor is a natrural approach to take to bring AI to your content.

Anaconda777’s picture

Anaconda777 CreditAttribution: Anaconda777 commented

Hi,

I have some thoughts related to AI and Drupal. Mostly services with Drupal are traditional content management related.
But what if Drupal site is a learning environment, or a social media platform? Then there might be different needs for AI integration.
Here are some what came into my mind as a "hobby" creator of a social learning environment with D9.

  1. Simple chat with AI for each users (students). Similar user experience is enough as now the openai website provides. I guess this is not yet done.
  2. Learning content creation. There are some learning related modules like Quiz and H5P, where a content creator could ask for AI to create a learning content from the topic X and ECA for example would insert that in the content type. For example create a multiple choice question set about topic X. Tried it, works.
  3. Grading automation for student created essays. For example if students are required to post an essay about topic X. Could AI create "pre grading" and grade the students submitted essays (nodes) automatically using a certain grading criterias? Now teacher needs to manually read all the essays and grade them, it is slow.

Then another thing is related to social media platform and content moderation. Could AI block users automatically who posts nude pictures of hate speech? Or some ECA action.. I guess with AI all this could be possible, but not yet a working solution with Drupal? And it might be expensive also if loads of content to analyze.
Then the last idea is related to user profiles, when there are loads of data like hobbies, interests or expertices. Could AI act like a personal coach, and align its output related to users profile? Also if a social media platform asks users mood, there could be analysis and automated "cheer up" responses from the AI.

ressa’s picture

ressa CreditAttribution: ressa at Ardea commented

I just heard about the new Open Source AI project Open Assistant.

It is still early, but the fact that it's free and Open Source is a big plus in my book.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented
Issue summary: View changes
  • Added info about #ai slack channel to the description.
  • Updated/simplified list of Drupal AI modules (still keeping it in the comment, because of frequent updates)
pinkonomy’s picture

pinkonomy CreditAttribution: pinkonomy commented

Is this possible with ChatGPT to have a content recommendation? How can this be achieved?

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

@pinkonomy I explored this successfully with movies.
You give AI a few movie examples, and GPT-4 recommends related movies. It does this with very high quality and explains the reasoning why something is a good choice.

So to create recommendation you give GPT:
1) previous examples what user liked
2) give list of options what could be recommended (content or titles from your site)
3) ask GPT to respond with recommendation in particular format (json maybe)

This can be done on a drupal side with prompt engineering tools. The beginning is here https://www.drupal.org/project/prompt and https://www.drupal.org/project/search_api_ai . Also I applied to Drupal innovation contest for kickstarter funding to improve some part of prompt engineering toolbox (results will be on Monday if selected).

Another part what is needed is larger context window to provide GPT with more information. I wrote about increasing of context window in this issue: #3358892: Fully fledged website and module development with Artificial Intelligence (summary: we need to wait until larger context window is accessible and affordable, but solutions can be programmed today)

Final part is caching and displaying recommendation results to the user. (this part unclear if modules exist. Maybe someone else can provide more info)

This is the simplest method and which will probably be most popular for small-average size sites. For large sites, solutions might be a bit more sophisticated (maybe again some of this https://www.drupal.org/project/search_api_ai)

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

This initiative participated in Drupal innovation contest. I tried to shape this project to cover most important aspects of AI and increase development resources by using AI itself.

My expectation was that half of other pitches will be about AI.
I thought - I will fill some stars (displayed in the pitch), and other winners will fill the other.
Sadly, there was only 1 AI pitch which itself lost. This of course is sad for me. I would like to learn some critique - what went wrong?

I hope there was something happening at the conference about AI development behind the scenes.

There is also possibility of funding still.

On the other hand, I don't know why almost nobody is interested in Chat UI for example (6 subscribers), I think this is a cool thing. I would say that issue is 5% done of what is possible. [Also, this is an opportunity for Drupal, but other projects will take the market share of this particular feature]

Also - I wonder how much new people Drupal attracts. Drupal was amazingly good 15-10 years ago. So much innovation was created in the short time. Right now, if I would be young, following AI developments, I would go on some other path. In other words, Drupal is probably not attracting people who want to build the next big thing. There is a lot of talk about ambitious site builders, but maybe Drupal is mostly at senior level now and still does not come back to its roots (yet?)

ressa’s picture

ressa CreditAttribution: ressa at Ardea commented

Maybe it was too theoretical?

I think you need to get those who haven't seen what it's capable of excited. If I were you, I would record a ChatGPT session, where you build a small Drupal module in ten minutes, which would normally take 1-2 hours.

Also, I think Drupal should lead with the choices, and focus on cooperating with AI Open Source projects such as Open Assistant and Hugging Face Chat UI.

I think it would be cool to cooperate with these projects, and build a prototype feeding their LLM's drupal.stackexchange.com, drupal.org, API.Drupal.org, etc. and iterate to build a cool and useful Drupal tool, on open source foundation, eventually better and freer than ChatGPT.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Thanks @ressa, it is always uplifting to see some other enthusiastic person.

I did build Drupal module in 5 hours, which would have taken 40 otherwise. And with a lot better quality. But chose not to show it, because I think audience would not have liked that. Because it can be seen as AI replacing jobs and I don't want pitch to be scary. I talked about adapting instead.

I decided to display something visual like changing theme colors, considering there is only 2.5 minutes (by the way, for some reason pitch was reduced to 2 minutes during conference by decision or accident, which was not good)

About open LLMs. Personally, I focus on practical application - it means creating Drupal features. It is a priority for me. Commercial models achieve that in my case. If I would do LLM, then features would be undone.

pinkonomy’s picture

pinkonomy CreditAttribution: pinkonomy commented

@mindaugasd
One question,did you use ChatGPT-4 for the 5 hours module creation or something else?

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

@pinkonomy I use custom primitive agents with GPT-4 model API. It is displayed in video, but I simplified it for video.
Another difference - I copy the code for now (while video shows automatic code editing prototype).
Also check out this example from 3 months ago about module creation in detail: #3336313-6: Use ChatGPT for solving Drupal issues to increase rate of development

cilefen’s picture

cilefen CreditAttribution: cilefen commented

Maybe people don’t want their jobs eliminated by robots.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

@cilefen that's the thing. My video is exactly a proposed solution to this problem: Adapting.
But there is another thing, not only jobs, but expertise. If someone prides themselves being an expert, then chatGPT takes that away.

ressa’s picture

ressa CreditAttribution: ressa at Ardea commented

@mindaugasd: You're welcome! I think we shouldn't fear this new invention, but look at it with an open mind, and see the dangers and down sides, as well as opportunities.

Interesting info about the module you built, and your 1:8 ratio in terms of saved time (5/40 hours) aligns with my estimate (10:60-120 minutes).

About replacing jobs, I get it. It's scary, but the taking over of tasks is going to happen. It's like horse carriage chauffeurs looking at cars in 1905, or table-tr-td-superheroes looking at CSS replacing tables twenty years ago: Why should we use that?! Or me, looking at Composer taking over Drupal ten years ago: "What's wrong with the tarball?" But things change.

As you write, we look at it from two different perspectives: I can see an almost immediate pay off using AI as a ressource for learning and building Drupal by training an LLM on Drupal ressources, and at the same time an opportunity to guide the development in an Open Source direction, for a less monopolistic, proprietary, black box world of Google, Facebook, Microsoft, Amazon, etc.

You are more into content creation. Both are very relevant. If someone sponsored you for working on a Drupal LLM ressource with Open Assistant/HuggingChat, would you be open for that?

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

@ressa If funding would be on the table to work on open LLM instead, possibly I would refuse (for above mentioned reasons) or agree after sorting out a lot of doubts.

Benefits of open LLMs and open source software cannot be directly compared. AI is very powerful tech. In the wrong hands, without safety rules, it can cause a lot of trouble. While OpenAI was created to make AI safe, many open LLMs are created to remove safety features. Overall, AI safety is super complicated subject. The hardest problem to solve ever, so I don't have clear position here.

About replacing jobs - it is a real issue too. It does not effect me personally, but I understand it can be devastating for people.

But AI is a natural progression of civilization, there is no alternative. It was always destined to happen sooner or later, and personally I want to participate in stearing a ship a little bit instead of ignoring the problem.

What is better direction - to be prepared for what is coming or unprepared? I choose option A.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Created a new issue to explore solving things we talked about "Learning about artificial intelligence and discussing its role within Drupal and Drupal community": #3365988: Learning about artificial intelligence and discussing its role within Drupal and Drupal community

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Drupal AI meetup

We will be having online meetup of Drupal AI community.
More information in this issue: #3365988-6: Learning about artificial intelligence and discussing its role within Drupal and Drupal community
Further meetups or updates will be announced in that issue as well.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Created a new issue to explore business side of AI building:
#3372329: Business case for AI building

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Created a new issue for studying AI safety
#3372687: Studying AI safety

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented

Significant milestone for Drupal AI is achieved today - AI prompt engineering module is released. It lets build prompts combining all kinds of Drupal data, it also acts as a database of prompts. For ultra flexibility - module has both "content" and "configuration" entities - letting AI builder decide what is best for their usecase.

On the other hand. module is a building block which has to be integrated with other AI modules to be valuable. Since module is new, there are no integrations at the moment.

Also I released or initiated many other AI modules in August. Take a look at AI initiative page to learn more.

Also see full and constantly updated list of all Drupal AI modules at the top of this issue.

mindaugasd’s picture

mindaugasd CreditAttribution: mindaugasd as a volunteer commented
Issue summary: View changes

Drupal has new great module ecosystem for building of AI tools https://www.drupal.org/project/ai_interpolator
This website https://workflows-of-ai.com/ have many videos showcasing how to use it and build AI features in the easiest most powerful way thus far. There is also an introductory video in depth: https://www.youtube.com/watch?v=Sw1Zk__M16c