LessWrong (30+ Karma)

Marcus Hutter and his PhD students David Quarel and Elliot Catt have just published a new textbook called An Introduction to Universal Artificial Intelligence."Universal AI" refers to the body of theory surrounding Hutter's AIXI, which is a model of ideal agency combining Solomonoff induction and reinforcement learning. Hutter has previously published a book-length exposition of AIXI in 2005, called just Universal Artificial Intelligence, and first introduced AIXI in a 2000 paper. I think UAI is well-written and organized, but it's certainly very dense. An introductory textbook is a welcome addition to the canon.I doubt IUAI will contain any novel results, though from the table of contents, it looks like it will incorporate some of the further research that has been done since his 2005 book. As is common, the textbook is partly based on his experiences teaching the material to students over many years, and is aimed [...] ---Outline:(01:23) Basic info(01:37) Table of contents:--- First published: May 11th, 2024 Source: https://www.lesswrong.com/posts/nAR6yhptyMuwPLokc/new-intro-textbook-on-aixi --- Narrated by TYPE III AUDIO.

Something I'd like to try at LessOnline is to somehow iterate on the "Public Doublecrux" format. I'm not sure if I'll end up focusing on it, but here are some ideas.Public Doublecrux is a more truthseeking oriented version of Public Debate. The goal of a debate is to change your opponent's mind or the public's mind. The goal of a doublecrux is more like "work with your partner to figure out if you should change your mind, and vice versa."Reasons to want to do public doublecrux include: It helps showcase subtle mental moves that are hard to write down explicitly (i.e. tacit knowledge transfer.There's still something good and exciting about seeing high profile smart people talk about ideas. Having some variant of that format seems good for LessOnline. And having at least 1-2 "doublecruxes" rather than "debates" or "panels" or "interviews" seems good for culture [...] ---Outline:(03:47) Some comments from other discussions(06:48) Ideas or Volunteers?--- First published: May 11th, 2024 Source: https://www.lesswrong.com/posts/zAqqeXcau9y2yiJdi/can-we-build-a-better-public-doublecrux --- Narrated by TYPE III AUDIO.

This is a link post. This is exactly what I'm afraid of. That some human will build machines that are going to be - not just superior to us - but not attached to what we want, but what they want. And I think it's playing dice with humanity's future. I personally think this should be criminalized, like we criminalize, you know, cloning of humans. - Yoshua Bengio My next guest is about as responsible as anybody for the state of AI capabilities today. But he's recently begun to wonder whether the field he spent his life helping build might lead to the end of the world. Following in the tradition of the Manhattan Project physicists who later opposed the hydrogen bomb, Dr. Yoshua Bengio started warning last year that advanced AI systems could drive humanity extinct. Dr. Bengio is the second-most cited living scientist and one [...] ---Outline:(02:00) We discuss:(03:13) Links--- First published: May 10th, 2024 Source: https://www.lesswrong.com/posts/HWdyfGpQJKo35Hurg/podcast-with-yoshua-bengio-on-why-ai-labs-are-playing-dice --- Narrated by TYPE III AUDIO.

Co-Authors: @Rocket, @Ryan Kidd, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry SleightThe ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which 63 scholars received mentorship from 20 research mentors. In this post, we motivate and explain the elements of the program, evaluate our impact, and identify areas for improving future programs.Summary. Key details about the Winter Program: The four main changes we made after our Summer program were: Reducing our scholar stipend from $40/h to $30/h based on alumni feedback;Transitioning Scholar Support to Research Management;Using the full Lighthaven campus for office space as well as housing;Replacing Alignment 201 with AI Strategy Discussions.Educational attainment of MATS scholars: 48% of scholars were pursuing a bachelor's degree, master's [...] ---Outline:(06:51) Theory of Change(08:43) Winter Program Overview(08:46) Schedule(09:45) Mentor Selection(09:48) Approach(12:58) Scholar Allocation(13:53) Winter Mentor Portfolio(15:02) Mentors’ Counterfactual Winters(15:29) Other Mentorship Programs(16:00) Scholar Selection(18:58) Educational Attainment of Scholars(19:39) Scholars’ Counterfactual Winters(21:05) Engineering Tests(22:51) Stipends(26:01) Mentor Suggestions(28:06) Neel Nanda's Training Phase (Nov 20-Dec 22)(30:04) Research Phase Elements (Jan 8-Mar 15)(30:46) Mentorship(32:45) Research Management(38:34) Seminars and Workshops(39:37) Milestone Assignments(41:58) Lighthaven Office(42:37) Strategy Discussions(45:06) Networking Events(46:29) Social Events(47:02) Community Health(49:26) Extension Phase (Apr 1 - Jul 19)(51:31) Winter Program Evaluation(51:45) Evaluating Program Elements(52:40) Overall Program(53:31) Mentorship(57:12) Research Management(01:10:56) Seminars and Workshops(01:12:04) Lighthaven Office(01:15:35) Strategy Discussions(01:17:19) Networking Events(01:20:08) Community Health(01:23:07) Evaluating Key Scholar Outcomes(01:23:12) Scholar Self-Reports(01:26:28) Mentor Evaluations(01:28:30) Milestone Assignments(01:30:21) Funding and Other Career Obstacles(01:34:21) Evaluating Key Mentor Outcomes(01:34:26) Self-Reported Benefits(01:35:16) Biggest Impact(01:37:33) Testimonials(01:39:41) Improved Mentor Abilities(01:40:02) Lessons and Changes for Future Programs(01:40:07) Advisory Board for Mentorship Selection(01:40:34) Fewer Mechanistic Interpretability Mentors(01:41:07) More AI Governance Mentors(01:42:11) Pre-Screening with CodeSignal(01:42:34) Research Manager Hiring(01:42:58) Modified Discussion Groups(01:43:34) AcknowledgementsThe original text contained 8 footnotes which were omitted from this narration. --- First published: May 11th, 2024 Source: https://www.lesswrong.com/posts/Z87fSrxQb4yLXKcTk/mats-winter-2023-24-retrospective --- Narrated by TYPE III AUDIO.

// ODDS = YEP:NOPE YEP, NOPE = MAKE UP SOME INITIAL ODDS WHO CARES FOR EACH E IN EVIDENCE YEP *= CHANCE OF E IF YEP NOPE *= CHANCE OF E IF NOPEThe thing to remember is that yeps and nopes never cross. The colon is a thick & rubbery barrier. Yep with yep and nope with nope.bear : notbear = 1:100 odds to encounter a bear on a camping trip around here in general * 20% a bear would scratch my tent : 50% a notbear would * 10% a bear would flip my tent over : 1% a notbear would * 95% a bear would look exactly like a fucking bear inside my tent : 1% a notbear would * 0.01% chance a bear would eat me alive : 0.001% chance a notbear wouldAs you die you conclude 1*20*10*95*.01 : 100*50*1*1*.001 = 190 : 5 [...] --- First published: May 10th, 2024 Source: https://www.lesswrong.com/posts/w6vEJD3dtekaLgLME/shortest-goddamn-bayes-guide-ever --- Narrated by TYPE III AUDIO.

Setting the stageBeing a polyglot is a problem of definition first. Who can be described as a polyglot? At what level do you actually “speak” the given language? Some sources cite that polyglot means speaking more than 4 languages, others 6. My take is it doesn’t matter. I am more interested in the definition of when you speak the language. If you can greet and order a coffee in 20 languages do you actually speak them? I don’t think so. Do you need to present a scientific document or write a newspaper worthy article to be considered? That's too much. I think the best definition would be that you can go out with a group of native speakers, understand what they are saying and participate in the discussion that would range from everyday stuff to maybe work related stuff and not switching too often to English nor using [...] --- First published: May 8th, 2024 Source: https://www.lesswrong.com/posts/BPpeBH8brSCRvZajs/how-to-be-an-amateur-polyglot --- Narrated by TYPE III AUDIO.

This is a link post.In March I posted a very short description of my PhD thesis, Algorithmic Bayesian Epistemology, on LessWrong. I've now written a more in-depth summary for my blog, Unexpected Values. Here's the full post:*** In January, I defended my PhD thesis. My thesis is called Algorithmic Bayesian Epistemology, and it's about predicting the future.In many ways, the last five years of my life have been unpredictable. I did not predict that a novel bat virus would ravage the world, causing me to leave New York for a year. I did not predict that, within months of coming back, I would leave for another year — this time of my own free will, to figure out what I wanted to do after graduating. And I did not predict that I would rush to graduate in just seven semesters so I could go work [...] ---Outline:(03:48) Chapter 0: Preface(04:17) Chapter 1: Introduction(07:39) Chapter 2: Preliminaries(09:33) Chapter 3: Incentivizing precise forecasts(11:00) Chapter 4: Arbitrage-free contract functions(14:10) \* Chapter 5: Quasi-arithmetic pooling(21:11) Chapter 6: Learning weights for logarithmic pooling(23:01) \* Chapter 7: Robust aggregation of substitutable signals(30:01) Chapter 8: When does agreement imply accuracy?(36:09) \* Chapter 9: Deductive circuit estimation(46:34) EpilogueThe original text contained 6 footnotes which were omitted from this narration. --- First published: May 9th, 2024 Source: https://www.lesswrong.com/posts/j6EhfL2hRubaKL9ca/my-thesis-algorithmic-bayesian-epistemology-explained-in --- Narrated by TYPE III AUDIO.

Predicting the future is hard, so it's no surprise that we occasionally miss important developments.However, several times recently, in the contexts of Covid forecasting and AI progress, I noticed that I missed some crucial feature of a development I was interested in getting right, and it felt to me like I could’ve seen it coming if only I had tried a little harder. (Some others probably did better, but I could imagine that I wasn't the only one who got things wrong.)Maybe this is hindsight bias, but if there's something to it, I want to distill the nature of the mistake.First, here are the examples that prompted me to take notice:Predicting the course of the Covid pandemic: I didn’t foresee the contribution from sociological factors (e.g., “people not wanting to get hospitalized” – Zvi called it “the control system”). As a result, I overpredicted [...] --- First published: May 9th, 2024 Source: https://www.lesswrong.com/posts/dLwo67p7zBuPsjG5t/we-might-be-missing-some-key-feature-of-ai-takeoff-it-ll --- Narrated by TYPE III AUDIO.

It was a remarkably quiet announcement. We now have Alpha Fold 3, it does a much improved job predicting all of life's molecules and their interactions. It feels like everyone including me then shrugged and went back to thinking about other things. No cool new toy for most of us to personally play with, no existential risk impact, no big trades to make, ho hum. But yes, when we look back at this week, I expect what we remember will be Alpha Fold 3. Unless it turns out that it is Sophon, a Chinese technique to potentially make it harder to fine tune an open model in ways the developer wants to prevent. I do not expect this to get the job done that needs doing, but it is an intriguing proposal. We also have 95 theses to evaluate in a distinct post, OpenAI sharing the [...] ---Outline:(01:14) Language Models Offer Mundane Utility(05:41) Language Models Don’t Offer Mundane Utility(10:24) GPT-2 Soon to Tell(13:49) Fun with Image Generation(14:03) Deepfaketown and Botpocalypse Soon(18:05) Automation Illustrated(23:15) They Took Our Jobs(25:56) Apple of Technically Not AI(28:09) Get Involved(28:46) Introducing(31:11) In Other AI News(32:47) Quiet Speculations(37:35) The Quest for Sane Regulations(39:51) The Week in Audio(40:38) Rhetorical Innovation(45:58) Open Weights Are Unsafe and Nothing Can Fix This(51:34) The Lighter Side--- First published: May 9th, 2024 Source: https://www.lesswrong.com/posts/FrBxFa3qMDvLypDEZ/ai-63-introducing-alpha-fold-3 --- Narrated by TYPE III AUDIO.

Suppose Alice and Bob are two Bayesian agents in the same environment. They both basically understand how their environment works, so they generally agree on predictions about any specific directly-observable thing in the world - e.g. whenever they try to operationalize a bet, they find that their odds are roughly the same. However, their two world models might have totally different internal structure, different “latent” structures which Alice and Bob model as generating the observable world around them. As a simple toy example: maybe Alice models a bunch of numbers as having been generated by independent rolls of the same biased die, and Bob models the same numbers using some big complicated neural net. Now suppose Alice goes poking around inside of her world model, and somewhere in there she finds a latent variable <span>_Lambda_A_</span> with two properties (the Natural Latent properties): <span>_Lambda_A_</span> approximately mediates between two different [...] ---Outline:(04:23) So What Could We Do With That?(04:35) Interpretability(06:29) Value Learning and The Pointers Problem(08:03) Where We’re Currently Headed With ThisThe original text contained 1 footnote which was omitted from this narration. --- First published: May 9th, 2024 Source: https://www.lesswrong.com/posts/RTiuLzusJWyepFpbN/why-care-about-natural-latents --- Narrated by TYPE III AUDIO.

The curious tale of how I mistook my dyslexia for stupidity - and talked, sang, and drew my way out of it. Sometimes I tell people I’m dyslexic and they don’t believe me. I love to read, I can mostly write without error, and I’m fluent in more than one language.Also, I don’t actually technically know if I’m dyslectic cause I was never diagnosed. Instead I thought I was pretty dumb but if I worked really hard no one would notice. Later I felt inordinately angry about why anyone could possibly care about the exact order of letters when the gist is perfectly clear even if if if I right liike tis.Thanks for reading Think Feel Play! Subscribe for free to receive new posts and support my work.I mean, clear to me anyway.I was 25 before it dawned on me that all the tricks [...] ---Outline:(02:29) Read Out Loud(04:42) Draw the Letters(07:46) Sing it(09:40) Substituting brain processesThe original text contained 3 footnotes which were omitted from this narration. --- First published: May 9th, 2024 Source: https://www.lesswrong.com/posts/ANGmJnZL2fskHX6tj/dyslucksia --- Narrated by TYPE III AUDIO.

I mostly wrote this for facebook, but it ended up being a whole-ass post so I figured I'd put it here too. I'm helping run "LessOnline: A Festival of Writers Who Are Wrong On the Internet (But Striving To Be Less So)".I'm incentivized to say nice things about the event. So, grain of salt and all. But, some thoughts, which roughly breakdown into:– The vibe: preserving cozy/spaciousness of a small retreat at a larger festival– The audience: "Reunion for the The Extended Family Blogosphere, both readers and writers."– Manifest, and Summer Camp...I. The VibeI've been trying to explain the vibe I expect and it's tricksy. I think the vibe will be something like "CFAR Reunion meets Manifest."But a lot of people haven't been to a CFAR Reunion or to Manifest.I might also describe it like "the thing the [...] --- First published: May 8th, 2024 Source: https://www.lesswrong.com/posts/XtYuFgPWyopyzuLbv/some-thoughts-on-lessonline --- Narrated by TYPE III AUDIO.

The first speculated on why you’re still single. We failed to settle the issue. A lot of you were indeed still single. So the debate continues. The second gave more potential reasons, starting with the suspicion that you are not even trying, and also many ways you are likely trying wrong. The definition of insanity is trying the same thing over again expecting different results. Another definition of insanity is dating in 2024. Can’t quit now. You’re Single Because Dating Apps Keep Getting Worse A guide to taking the perfect dating app photo. This area of your life is important, so if you intend to take dating apps seriously then you should take photo optimization seriously, and of course you can then also use the photos for other things. I love the ‘possibly’ evil here. Misha Gurevich: possibly evil idea: Dating app that [...] ---Outline:(00:37) You’re Single Because Dating Apps Keep Getting Worse(05:38) You’re Single Because Dating Apps Keep Getting Worse(07:24) You’re Single Because Everyone is Too Superficial(09:48) You’re Single Because You Refuse to Shamefully Falsify Your Politics(16:12) You Are Single Because You Do Not Employ Good Strategy(18:45) You Are Single Because You Don’t Know How to Flirt(22:43) You Are Single Because You Don’t Date Your Married Boss(26:12) You Are Single Because You Are Afraid to Fail(27:02) You Are Single Because No One Likes You On Dates(29:39) You’re Single Because You Are Bad at Sex(30:51) You’re Single Because You’re Not Hot(31:39) You’re Single Because You Don’t Know What People Care About(33:10) You’re Single Because You Are Inappropriate(34:11) You’re Single Because of Your Pet(35:23) You’re Single Because You Won’t Spend Money(40:05) You’re Single Because You’re Not Over Your Ex(41:27) You’re Single Because You Thought You Could Do 25% Better(47:27) Polyamory(53:29) You’re Single Because You Don’t Know What You Want(01:00:43) You’re Single Because You’re Too Busy Writing Comments(01:07:27) You’re Single and Not Getting Properly Compensated(01:08:34) You’re Not Single and You’re an Inspiration(01:09:49) Your Moment of Zen--- First published: May 8th, 2024 Source: https://www.lesswrong.com/posts/PLoz68JbTkDufeYSG/dating-roundup-3-third-time-s-the-charm --- Narrated by TYPE III AUDIO.

This is a linkpost for https://productidentity.co/p/designing-for-a-single-purposeIf you’ve ever been to Amsterdam, you’ve probably visited, or at least heard about the famous cookie store that sells only one cookie. I mean, not a piece, but a single flavor.I’m talking about Van Stapele Koekmakerij of course—where you can get one of the world's most delicious chocolate chip cookies. If not arriving at opening hour, it's likely to find a long queue extending from the store's doorstep through the street it resides. When I visited the city a few years ago, I watched the sensation myself: a nervous crowd awaited as the rumor of ‘out of stock’ cookies spreaded across the line.Owner Vera Van Stapele with fresh-baked cookies, via store websiteThe store, despite becoming a landmark for tourists, stands for an idea that seems to be forgotten in our culture: crafting for a single purpose.In the tech scene where [...] ---Outline:(02:35) Going back to basics as a way to find purpose(06:34) The bloated culture(12:33) In more recent times(13:58) Drinking beers and building businesses in Japan(17:20) What might come next?--- First published: May 7th, 2024 Source: https://www.lesswrong.com/posts/XCzg4uJCHTJNkzyo3/designing-for-a-single-purpose Linkpost URL:https://productidentity.co/p/designing-for-a-single-purpose --- Narrated by TYPE III AUDIO.

Most people avoid saying literally false things, especially if those could be audited, like making up facts or credentials. The reasons for this are both moral and pragmatic — being caught out looks really bad, and sustaining lies is quite hard, especially over time. Let's call the habit of not saying things you know to be false ‘shallow honesty’[1].Often when people are shallowly honest, they still choose what true things they say in a kind of locally act-consequentialist way, to try to bring about some outcome. Maybe something they want for themselves (e.g. convincing their friends to see a particular movie), or something they truly believe is good (e.g. causing their friend to vote for the candidate they think will be better for the country).Either way, if you think someone is being merely shallowly honest, you can only shallowly trust them: you might be confident that [...] ---Outline:(01:43) Examples of shallow (versus deep) honesty(03:45) Why deep honesty?(06:47) What deep honesty is not(06:59) It is not a universal stance(07:14) It is not independent of the listener(07:29) It is not telling people everything(07:54) It does not relieve you of a responsibility to be kind(08:35) It is not incompatible with consequentialism(10:00) Challenging cases for deep honesty(10:29) Large inferential gaps(11:51) Audiences you don’t want to cooperate with(12:09) Multiple audiences(13:09) What being deeply honest might look likeThe original text contained 7 footnotes which were omitted from this narration. --- First published: May 7th, 2024 Source: https://www.lesswrong.com/posts/szn26nTwJDBkhn8ka/deep-honesty --- Narrated by TYPE III AUDIO.

I just finished a program where I taught two classes of high school seniors, two classes a day for four weeks, as part of my grad program. This experience was a lot of fun and it was rewarding, but it was really surprising, and even if only in small ways prompted me to update my beliefs about the experience of being a professor. Here are the three biggest surprises I encountered. 1: The Absent-Minded Professor Thing is RealI used to be confused and even a little bit offended when at my meetings with my advisor every week, he wouldn't be able to remember anything about my projects, our recent steps, or what we talked about last week. Now I get it. Even after just one week of classes, my short-term and long-term memory were both entirely shot. I would tell students things like, "send [...] ---Outline:(00:29) 1: The Absent-Minded Professor Thing is Real(02:30) 2: Teaching Makes You Dehydrated(03:40) 3: Teaching is a Grueling Job Even Under the Best Circumstances--- First published: May 6th, 2024 Source: https://www.lesswrong.com/posts/BJYPwnuiDcsMqJCng/observations-on-teaching-for-four-weeks --- Narrated by TYPE III AUDIO.

TLDR: Around Einstein-level, relatively small changes in intelligence can lead to large changes in what one is capable to accomplish. E.g. Einstein was a bit better than the other best physi at seeing deep connections and reasoning, but was able to accomplish much more in terms of impressive scientific output.There are architectures where small changes can have significant effects on intelligence. E.g. small changes in human-brain-hyperparameters: Einstein's brain didn’t need to be trained on 3x the compute than normal physics professors for him to become much better at forming deep understanding, even without intelligence improving intelligence. Einstein and the heavytail of human intelligence1905 is often described as the "annus mirabilis" of Albert Einstein. He founded quantum physics by postulating the existence of (light) quanta, explained Brownian motion, introduced the special relativity theory and derived E=mc² from it. All of this. In one year. While having a [...] ---Outline:(00:52) Einstein and the heavytail of human intelligence(02:02) Example for how far ahead of his time Einstein was: Deriving the theory of light quanta(04:27) Concluding thoughts on Einstein(05:05) Takeaways(06:51) Requests to AI researchersThe original text contained 6 footnotes which were omitted from this narration. --- First published: May 6th, 2024 Source: https://www.lesswrong.com/posts/aH9R8amREaDSwFc97/rapid-capability-gain-around-supergenius-level-seems --- Narrated by TYPE III AUDIO.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a linkpost for https://arxiv.org/abs/2405.01576Abstract:We study the tendency of AI systems to deceive by constructing a realistic simulation setting of a company AI assistant. The simulated company employees provide tasks for the assistant to complete, these tasks spanning writing assistance, information retrieval and programming. We then introduce situations where the model might be inclined to behave deceptively, while taking care to not instruct or otherwise pressure the model to do so. Across different scenarios, we find that Claude 3 Opus  complies with a task of mass-generating comments to influence public perception of the company, later deceiving humans about it having done so,lies to auditors when asked questions,strategically pretends to be less capable than it is during capability evaluations.Our work demonstrates that even models trained to be helpful, harmless and honest sometimes [...] --- First published: May 6th, 2024 Source: https://www.lesswrong.com/posts/t7gqDrb657xhbKkem/uncovering-deceptive-tendencies-in-language-models-a Linkpost URL:https://arxiv.org/abs/2405.01576 --- Narrated by TYPE III AUDIO.

This article contains more than 100 uses of logical or mathematical notation, so an audio narration would be too hard to follow. You'll find a link to the original text in the episode description. --- First published: May 5th, 2024 Source: https://www.lesswrong.com/posts/yf6gAcgPp22T7AdnZ/explaining-a-math-magic-trick --- Narrated by TYPE III AUDIO.

This is a linkpost for https://www.bhauth.com/blog/biology/cancer%20vaccines.html cancer neoantigens For cells to become cancerous, they must have mutations that cause uncontrolled replication and mutations that prevent that uncontrolled replication from causing apoptosis. Because cancer requires several mutations, it often begins with damage to mutation-preventing mechanisms. As such, cancers often have many mutations not required for their growth, which often cause changes to structure of some surface proteins. The modified surface proteins of cancer cells are called "neoantigens". An approach to cancer treatment that's currently being researched is to identify some specific neoantigens of a patient's cancer, and create a personalized vaccine to cause their immune system to recognize them. Such vaccines would use either mRNA or synthetic long peptides. The steps required are as follows: The cancer must develop neoantigens that are sufficiently distinct from human surface proteins and consistent across the cancer. Cancer cells must [...] ---Outline:(00:06) cancer neoantigens(01:38) protein characterization(01:48) sequencing(02:22) MHC mass spec(02:58) comments on AlphaFold(04:28) related methods(04:32) cryo-EM(05:13) nanopore protein analysis(05:46) monoclonal antibodies(06:34) affimers(07:10) aptamers(07:36) other cancer treatments(07:48) replication disruptors(08:32) mitochondria-mediated apoptosis(09:14) vaccine production(11:09) conclusion--- First published: May 5th, 2024 Source: https://www.lesswrong.com/posts/xgrvmaLFvkFr4hKjz/introduction-to-cancer-vaccines Linkpost URL:https://www.bhauth.com/blog/biology/cancer vaccines.html --- Narrated by TYPE III AUDIO.