Random Observation/Comment #813: On one hand, I hope the open source LLM battle wins. On the other, I think we need some guardrails or we’ll accidentally connect our services to Skynet and lead to our doom.
//Generated on Midjourney using a pan left and right feature. I used this in an internal company presentation showing the launch of Claude 2 and upgrades to Bard.
Why this list?
AI chatbots are all the rage. I think we’re in the next evolution of the web, which is an AI-enabled web with all human organic content solidified in 2023. How can we trust any of the words or images are genuine and not augmented? Similar to ads and promoted posts, we will be trained to doubt the skill and reality of the new internet.
- Base chat interface with new threads trained on Sept 2021 data. Each thread maintains the knowledge and context of the conversation. Thread names are summarized by ChatGPT.
- Sharing chat links – You can also share your specific chat thread instance with others to view.
- GPT4 and beyond – More powerful models with GPT5 that’s trained with fewer parameters
- ChatGPT with internet search capabilities – This was briefly available with Bing, but people used this to get past pay walls or directly query archives
- ChatGPT plugins and plugin store – This really opens up the possibility of anyone writing plugins and apps. The search function for this is a bit crap though.
- Self executing code – ChatGPT can write python code and execute it within a thread instance. This is incredibly powerful.
- Code interpreter with Upload files – This latest feature let’s you upload and read files. The most common examples are around csv data analysis and trend questions.
- Custom instructions – Specify the input context and output style for all chats. This just launched so I need to explore it more for superpowered responses.
- Detailed sandbox with variations on responses – The playground is actually much easier to use for writing than just chats
New ChatGPT Feature Ideas
Specific to LLMs and their manifestation for users, here’s my guess at some of the features that could rollout for ChatGPT. Some of these are already features in popular extensions, bard, Claude 2, and Poe.
- Increased input window size – Claude 2 has a 100k input window or 75k word memory of context, which is perfect for reading and processing long PDFs.
- Intuitive private vector database training – Personalized training on your material like books, Facebook posts, or Instagram photos. This could be dangerous if we’re connecting to public sources. I imagine there would be privacy concerns if you’re hooking up social networks.
- Cross thread commands for long term memory – Instead of a separate custom instruction, it would be awesome to specify a command that saved certain information that would be relevant for all new threads. This means the custom instruction page is a summary written by ChatGPT of the things we talked about rather than me adding information to all future prompts in a input/output style way. For example, if I can write “Remember that my name is Clemens. Please call me Clemens.” then a new thread should just say “Welcome back, Clemens. How can I help you, today?”
- Integration with slack, discord, and web widgets – The power behind this is really the transparent multi-user interaction and therefore training of AI agents and experts within a company. If this works then a company’s valuation can include the amount of time/hours of personalized training of these bots. Where’s my onboarding training bot? This can also then manifest itself as an external facing service (although, people would totally try to backdoor the AI bot for hints on roadmap or IP).
- ChatGPT for enterprise (company wide historian and advisor) – This would be a privately hosted instance that would learn all docs, slack channels, notion pages, and product code so that you can ask it anything about your product for internal training.
- ChatGPT project manager – Integration of all data created by the company including Jira tickets and code commits would mean this bot can give a summary of how the project is advancing. This would reduce the middle management role and focus on “Operational Happiness”
- Persistent writing styles and personalities – Ask ChatGPT to save my personal writing style after it has read my books and material. Then I can invoke a request to write in the style of “Clemens” or write in my own style.
- Conversational mode – I think you can include this in the prompt, but I’d be interested to have a thread with a more conversational chatbot that responds with more questions and prompts. Who needs a therapist when I have a chatbot?
- Private personalized GPT instances – Making a Clembot that I can use as an interface to all digital things would be awesome. This would mean my Clembot would be the base layer of connectivity across devices and APIs into different social networks.
- Recursive goal oriented initiatives – AutoGPT has this and I think the code interpreter upgrade does a plan with additional problem solving in order to use other library alternatives and consider how to complete the project. It also asks for more information and preferences.
- Access to hosting other open source models – Poe provides access to hosted models and passthrough for GPT3.5 and GPT 4. I also like the LLM comparison library on nat.dev
- Embed ChatGPT into my WordPress – I would want self-contained access to speaking with the author or blogger. Maybe there’s also a YouTube plugin so you can interact with your social version of yourself.
- Image recognition and interpretation – Bard does this with Google Lens connectivity. It can interpret new pieces and even recommend how to remake the image with a prompted generative AI tool.
- Directly embedded image generation with Dall-E – I’m surprised this isn’t out already, but it would be interesting to directly generate images and previews in the chat. I think this is more of a bandwidth issue.
- Fact checking and references with follow-ups – I love how this is done with Perplexity.ai and the interface gives more followup questions
- Recommended or popular prompts – I imagine this is similar to type ahead in search. I do like how Midjourney and other image generating apps provide open examples of prompts to generate photos.
- Drop downs for voice and style of writing in the response – There are some extensions that already has this available.
- Mobile app with access to mobile integration – There could be a biometric lock or it can access location data to provide restaurant recommendations nearby with my particular taste.
- Streaming latest news and topics of interest – It would be cool to create a bot that listens to an RSS feed and summarizes my emails or followed news channels. This could be great for market competitor analysis newsletters.
- Video transcriptions, subtitles, and summaries – I believe the Vox plugin does this already, but if could upload a video mp4 file then the AI could add subtitles to it automatically and let you distribute to different mediums.
- Voice integration to read responses – This could be a direct integration with the voices in Elevenlabs. Bard has implemented this already and it reads in 40+ languages.
- Login with Google and give access to using your own data across Google suite applications – Deep learning of Clembot is scary, but also cool because it could be my external facing worker. A future business could be managing my custom-trained and deployed AI agents embedded into team slacks. This is how I can scale to contribute to all the DAOs I’ve joined. Google is likely doing this with Duet AI.
- Login with Google and allow for basic functions like the ones in IFTTT – If you look at the available APIs, this could allow for a lot of access to calendars, emails, texts, phone calls, etc
- Integration with Google assistant, Siri, or Amazon Alexa – Replace my voice assistant with GPT4 with internet capabilities and access to creating calendars. I would want more conversations than services.
- Live interactive conversation mode – Voice to Text to LLM to Text response to Voice with streaming. This was done with Nvidia’s gaming plugin for AI NPCs. It would be cool to do this for translators or travel assistants.
- Proactive learning off of a goal in a subject – Within BabyGPT there was a curiosity and engineering mindset for finding different tasks to complete a goal. I think this is a super power.
- ChatGPT social media manager – I don’t think this will replace Hubspot, but it would be nice to get a chat interface for reporting and summaries. It could also include features with cross posting and ability to express ideas differently to better know my audience.
- Notifications – I’m already returning to the application to ask questions, but I would be even more addicted if ChatGPT was running jobs in the background and then notifying me when there are major changes in the stock market or news that would be relevant based on the previous chat topics that I’ve asked. This is more about the longer memory allocation for users to make the bot more personalized.
- Metaverse and VR integration – If Llama 2 created an NPC in my Meta Quest home then it could be pretty fun. I’m already using voice activation for the VR space instead of typing so I could talk to my GPT chatbot.
- Home automation setup and access – Time to build a full Jarvis. I would love to see the next phase installations of security tools and IOT devices connected with the conversational analysis.
~See Lemons Use ChatGPT