If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though.

There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too. The OpenAI demo passes every 50th frame of a ~600 frame video (20s at 30fps).

EDIT: As noted in discussions below, Gemini 1.5 appears to take 1 frame every second as input.

Most notably, at 1FPS the GPT-4V API errors out around 3-4 mins, while 1.5 Pro supports upto an hour of video inputs.

The gemini 1.5 pro uses about 258 tokens per frame (2.8M tokens for 10856 frames).

Are those comparable?

For structured data extraction I also like not having to run pseudo-OCR on hundreds of frames and then combine the results myself.

https://developers.googleblog.com/2024/02/gemini-15-availabl...

"Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..."

But it's very likely individual frames at 1 frame/s

https://storage.googleapis.com/deepmind-media/gemini/gemini_...

"Figure 5 | When prompted with a 45 minute Buster Keaton movie “Sherlock Jr." (1924) (2,674 frames at 1FPS, 684k tokens), Gemini 1.5 Pro retrieves and extracts textual information from a specific frame in and provides the corresponding timestamp. At bottom right, the model identifies a scene in the movie from a hand-drawn sketch."

I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that.

UPDATE ... posted that before you edited your post to link to the Gemini 1.5 report.

684,000 (total tokens for the movie) / 2,674 (their frame count for that movie) = 256 tokens - which is about the same as my 258 tokens for a single image. So I think you're right - it really does just split the video into frames and process them as separate images.

> The model processes videos as non-contiguous image frames from the video. Audio isn't included. If you notice the model missing some content from the video, try making the video shorter so that the model captures a greater portion of the video content.

> Only information in the first 2 minutes is processed.

> Each video accounts for 1,032 tokens.

That last point is weird because there is no way a video would be a fixed amount of tokens and I suspect is a typo. The value is exactly 4x the number of tokens for an image input to Gemini (258 tokens) which may be a hint to the implementation.

I'm definitely not a fan of these severely hamstrung by default models. Especially as it seems to be based on an extremely puritan ethical system.

Also agree that it’s mostly policed by American companies who follow the American culture of “swearing is bad, nudity is horrible, some words shouldn’t even be said”

I'd like control over what the guard rails do. I'd still use them under most circumstances, there's things I definitely do not want to generate, but if a word filter is getting in my way I'd like the ability to get rid of it.

Likewise it won't generate passive-aggressive answers meant for comedic reasons.

I hate having to negotiate with AI like it's a difficult child.

What definition of 'rhymes' are you using here?

Is it sexual? Is it alcohol? Is it violence? All of the above?

For example, good luck ever actually processing art content with that approach. Limiting everything to the lowest common denominator to avoid stepping on anyone's toes at all times is, paradoxically, a bane on everyone.

I believe we need to rethink how we deal with ethics and morality in these systems. Obviously, without a priori context every human, actually every living being, should be respected by default and the last thing I would advocate for is to let racism, sexism, etc. go unchecked...

But how can we strike a meaningful balance here?

[0] https://www.nytimes.com/2022/11/19/style/tiktok-avoid-modera...

Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably.

Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this pattern" or "i noticed you have signs of this diagnosis from the way you move your mouse and consume content, may i recommend this lifestyle change".

I wonder how long before something like that is feasible, ie a model you install that is constantly updated, but also constantly merged with world data so it becomes more intelligent on two fronts, and can follow as hardware and software advances over the years.

Such a model would be dangerously valuable to corporations / bad actors as it would mirror your psyche and remember so much about you - so it would have to be running with a degree of safety i can't even imagine, or you'd be cloneable or loose all privacy.

It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

For example, consider the classic situation of accidentally giving someone the same Christmas that you did a few years back. A sufficiently powerful personal LLM that 'remembers everything' could absolutely help with that (maybe even give you a nice table of the gifts you've purchased online, who they were for, and what categories of items would complement a previous gift), but only if it can practically store that memory for a multi-year time period.

Just did the second phase of using ocrmac (vision kit cli on GitHub) that extracts text and dumps it in a SQLite with FTS5.

It’s simplistic but does the job for now.

I looked at reducing storage requirements by using image magik to only store the difference between images - some 5 min sequence are essentially the same screen - but let that one go.

Well, that's basically how video codecs work... So might as well just find some codec params which work well with screen capture, and use an existing encoder.

And if you want to use the data for LLM only, then you don't need to store the screenshots at all. Then it's ~ 15MB a month

Cries in MacBook Pro

Shipped to the UK for me added a bit to the overall price with shipping and import duty but it was still better value for money and hugely reliable brand than anything I could have bought domestically.

“Purchased socks from Amazon for $10 on 12/4/2024 at 5:04PM, shipped to Mom, 1600 Pennsylvania Av NW, Washington DC 20500, order number 1463355337

Probably stored in a vector DB for RAG.

No mention of using any LLMs in there at all which is how you are presenting it in your comment here.

Any idea what would make you feel safe? 3rd party verification? I had it verified and published by the Microsoft Store. I feel eventually it all comes down to me being a decent person.

https://github.com/Zetaphor/screendiary/

This is my application, it does not have AI running on top.

And then announcing "I can do your job now. You're fired."

As in human marriage, there should be a law that said your AI-companion cannot be compelled to testify against you :-)

It can also be local to my own computer. People do write software while they're away from work.

You humans think that the AI will have someone in charge of it. Look, that's a thin layer that can be eliminated quickly. It's like when you build a tool that automates the work of, say, law firms but you don't want law firms getting mad that you're giving it away to their clients, so you give it to the law firms and now they secretly use the automating software. But it's only a matter of time before the humans are eliminated from the loop:

https://www.youtube.com/watch?v=SrIf0oYTtaI

The employee will be eliminated. But also the employer. The whole thing can be run by AI agents, which then build and train other AI agents. Then swarms of agents can carry out tasks over long periods of time, distributed, while earning reputation points etc.

This movie btw is highly recommended, I just can't find it anywhere anymore due to copyright. If you think about it, it's just a bunch of guys talking in rooms for most of the movie, but it's a lot more suspenseful than Terminator: https://www.youtube.com/watch?v=kyOEwiQhzMI

And since that has to be the case, we might as well find fun and profit wherever we can -- while we still can.

If that means that my desktop robot is keeping tabs on me while I write this, then so be it as long as I get some short-term gain. (There can be no long-term gain.)

It probably breaks the spirit of the employment contract just as hard, but it's essentially undetectable for the work computer.

https://philosophy.williams.edu/files/Egan-Learning-to-Be-Me...

And what is the likelihood of that "of course" portion actually happening? What is the business model that makes that route more profitable compared to the current model all the leaders in this tech are using in which they control everything?

I understand the actual screen recordings don't leave your machine, but that just creates a catch-22 of what does. Either the text based summaries of those recordings are thorough enough to still be worthy of privacy or the actual answers you get won't actually include many details from those recordings.

[1] - https://help.rewind.ai/en/articles/7791703-ask-rewind-s-priv...

Of course taking notes and bookmarking things is possible, but you can't include everything and it takes a lot of discipline to keep things neatly organized.

So we take it for granted that every once in a while we forget things, and can't find them again with web searching.

But with the new LLMs and multimodal models, in principle this can be solved. Just describe the thing you want to recall in vague natural language and the model will find it.

And this kind of retrieval is just one thing. But if it works well, we may also grow to rely on it a lot. Just as many who use GPS in the car never really learn the mental map of the city layout and can't drive around without it. Yeah, I know that some ancient philosopher derided the invention of books the same way (will make our memory lazy). But it can make us less capable by ourselves, but much more capable when augmented with this kind of near-perfect memory.

What they fill their homes with are definitely microphones, with the google assistant and amazon echos.

Back then, I used on device OCR and then sent the text to gpt. I’ve been wanting to re-do this with local LLMs

This is already possible to implement today, so it's very likely that we'll all have our own personal AIs that know us better than we do.

When you consider it, we aren't very far away from that at all.

I bet meta is thinking of doing this with quest once the battery life improves.

https://rewind.ai/pendant

>Our service, Ask Rewind, integrates OpenAI’s ChatGPT, allowing for the extraction of key information from your device’s audio and video files to produce relevant and personalized outputs in response to your inputs and questions.

The rest of the company has info on their other free/paid offerings and the split is pretty closely "what do we need to pay for an API to do vs do locally".

Again, I'm not associated with them, but that was my expectation after looking at it.

Only for people who'd pay for that.

Free users would become the product.

The x230 is a popular and interesting thinkpad with a powerful i5 processor suitable for today’s needs.

The T60 can also suit your needs and is one of the last IBM thinkpads. It featured the latest Intel mobile processor at the time of its release.

If you want the most powerful thinkpad the T440p is sure to suit you perfectly without leaving your morals behind.

...accompanied by the wrath of countless others discouraging you from trying to fork if you even so much as give slight indications of wanting to do so, and then when you do, they continue to spread FUD about how your fork is inferior.

I've seen plenty of discussions here and elsewhere where the one who suggests forking got a virtual beating for it.

Machine learning is fun and ultimately it doesn't require a lot of code. If people have the compute, open source maintainers will have the interest to exploit it due to the high coolness-to-work-required ratio.

Clippy + high IQ: red flag, right here

Clippy + high IQ + intimate knowledge of you: do you seriously want that? Why?

It can do as much as it wants in the background turning that into timely and non-intrusive actionable behaviours is extremely challenging.

I spent a long time thinking about a global notification consumption system that would parse all desktop, mobile, email, slack, web app, etc notifications into a single stream and then intelligently organizes it with adaptive timing and focus streams.

The cross platform nature made it infeasible but it was a fun thought experiment because we often get repeated notifications on every different device/interface and most of the time we just zone it out cuz it’s overload.

Adding a new nanny to your desktop is just going to pile it on even more so you have to be careful.

I don't know if it's public yet, but they sent me this video with the invite: https://youtu.be/dXvhGwj4yGo

He was talking about eventually having an agent that watches your screen and remembers what you do across all apps, and can store it and share it with you team.

So you could say “how does my teammate run staging builds?” or “what happened to the documentation on feature x that we never finished building”, and it’ll just know.

Obviously that’s far away, and it was just the ramblings of excited founder, but it’s fun to think about. Not sure if I hate it or love it lol

Speech-to-text works. OCR works. LLMs are quite good at getting the semantics of the extracted text. Image understanding is pretty good too already. Just with the things that already exist right now, you can go most of the way.

And the CCTV cameras will also all be processed through something like it.

A use that seems scientifically possible but technically difficult would be to have an LLM help you engage in essentially immersion learning. Set up something like a pihole, but instead of cutting out ads it intercepts all the content you're consuming (webpages, text, video, images) and translates it to the language you're learning. The idea would be that you don't have to go out and find whole new sources of language to set yourself with a different language's information ecosystem, you can just press a button and convert your current information ecosystem to the language you want to learn. If something like that could be implemented it would be incredibly valuable.

I'll probably get pushback on the quality of things like auto-generated subtitles, but I did the above to watch and understand a long interview I was interested in but don't possess skill in the language they were using. That was to turn the content into something I already know, but I could do the reverse and turn English content into French or whatever I'm trying to learn.

Imagine if it was wrong about something. But every time you tried to submit the bug report it disables your arms via Nueralink.

When it can gently nag you via a phone app to study and have a fake zoom call with you to be more engaging it feels like that could get much better results than the current online courses.

Could you elaborate on what was useless about the screen capture timeline?

1. I use a limited set of tools (Slack, GitHub, Linear, email), each providing good search capabilities.

2. I can remember things people said, and I said, in a fairly detailed way, and accessing my memory is faster than using a UI.

Other minor factors include: I take screenshots judiciously (around 2500-3000 per year) and bookmark URLs (13K URLs on Pinboard). Rewind did not convince me that it was doing all of this twice as well.

Saw this yesterday, 1M context window but haven't had any time to look into it, just an example new developments happening every week:

https://www.reddit.com/r/LocalLLaMA/comments/1as36v9/anyone_...

But yeah, dystopia is right down the same road we're all going right now.

The Four, especially with its AI, is going to control the market in ways that will have a deep impact on government and society.

As technological society advances the threshold to enter the market with anything not completely laughable becomes exponentially harder, only consolidating old money or the already established right?

What i found so amazing about the early internet, or even just the internet 2.0 was the possibility to create a platform/marketplace/magazine or whatever, and actually have it take off and get a little of the shared growth.

But now it seems all growth has become centralised to a few apps and marketplaces and the barrier to entry is getting harder by the hour.

Ie. being an entrepreneur is harder now because of tech and market consolidation. But potentially mirrored in previous eras like the industrialisation - i'm just not sure we'll get another "reset" like that to allow new players.

Please someone explain how this is wrong and there's still hope for the tech entrepreneurs / sideprojects!

At that point, why pay you at all?

https://www.reddit.com/r/memes/comments/bb1jq9/clippy_is_qui...

[1] https://superb-owl.link/shapes-of-stories/#1513

Good gracious could I please just use an unfiltered model? Or maybe one which isn’t so sensitive?

I colloquially used the word "hack" when trying to write some code with ChatGPT, and got admonished for trying to do bad things, so uncensoring has gotten interesting to me.

I'd rather avoid sharing my thoughts and interests with this Borg-like entity.

Whether you go with microsoft, google, meta, or whatever apple will come up with, it feels like a case of "stay out, or make a pick and stick to it".

I know some have different feelings regarding this or that company that is "better" or "worse", but the reality of it is they're not, and even if they were you don't know where they will be in ten years, and they will still have your data then.

Nonetheless I'd be really worried if you were working on a startup whose training process started with "We'll just scrape YouTube because that is for all intents and purposes public data".

So if your job means you use a specific OS with a specific Office Suite in the cloud and that office suite in the cloud incorporate AI and you only get half the features available if you don't consent, you as an employee end up kind of forced to consent anyway, GPDR or not.

[1]: https://proton.me/blog/eu-council-encryption-vote-delayed

That being said there is a difference between the US and the EU in regards to how these things are approached. Where the US is more likely to let private companies destroy privacy while keeping public agencies leashed it’s the opposite in Europe. Truth be told, it’s not like the US initiatives are really working since agencies like the NSA seem to blatantly ignore all laws anyway, which cause some scandals here in Europe as well. In Denmark our Secret Police isn’t allowed to spy on us without warrants, but our changing governments has had different secret agreements with the US to let the US monitor our internet traffic. Which is sort of how it is, and the scandal isn’t so much that, it’s how our Secret Police is allowed to get information about Danish citizens from the NSA without warrants, letting our secret police spy on us by getting the data they aren’t allowed to gather themselves from the NSA who are allowed to gather it.

Anyway, it’s a complicated mess, and you have so many branches of the bureaucracy and so many NGOs pulling in different directions that you can’t say that the EU is pro or anti privacy the way you want to. Because it’s both of those things and many more at the same time.

I think the only thing the EU unanimously agrees on (sort of) is to limit private companies access to citizen privacy data. Especially non-EU organisations. Which is very hard to enforce because most of the used platforms and even software isn’t European.

I am not fine with government using the data to police me. Already in most countries, governments are putting people in jail because of things like hate speech where are the laws are really vague.

I think everybody (private companies, government, individuals) will try to influence and will affect your personal life. What I am worried about is who has the most efficient way to influence a lot the average person - because that entity can control on long term a lot more.

My impression is that in the European Union - due partially to a complex system - is harder for any particular actor to do much on its own (even the example with Denmark secret service asking NSA for data about citizens - I guess it is harder for them to do that rather than just get directly the data).

So what I am afraid is focused and efficient entities having the data, hence I am more afraid of private companies (which are focused and sometimes efficient) rather than governments.

Are you saying influencing life through ads and putting me in jail have similar effect on me? If you combine all laws of my country I am pretty sure I would have broken few unintentionally. If government wants to just put me in jail they could retroactively find any of my past instance if they have the data. This is not some theoretical thing, but something the thing that happens with political dissidents all the time.

And there doesn't appear to be examples of those 6 countries imprisoning people for those laws.

That's true of most places. We should applaud the EU's human rights court for leading the way by banning this behavior: https://www.eureporter.co/world/human-rights-category/europe...

Plenty of companies have been shoving all the unstructured data they have about you and your friends into a big neural net to predict which ad you're most likely to click for a decade now...