His Divine Shadow wrote:Would it be safe to build a connectivist AI after "safe" AI's have been developed, that is post-singularity, just for intellectual curiosity.
Probably. A connectionist AI is going to be inherently less efficient and hence less intelligent, if the hardware is anything like current computers. For the connectionist one to be a threat the existing AGIs would have to be deliberately ignoring it (or unaware of it) for long enough for it to build up a major resource base (enough to overcome its intelligence deficit, assuming it doesn't restructure itself to drop its original design quirks). In most sci-fi situations, all that is unlikely; you could safely mess about with utterly bizarre cognitive architectures, with a benevolent normative AGI around to monitor and where necessary contain the results.
I am thinking maybe a safe AI could keep up with any potentially hostile connectivist AIs that would develop. That could then yield insight into how connectivist AI works and give deeper understanding of such a system and how to make it safer, or is that just fundamentally impossible with a connectivist design even for a superintelligent AI?
Well, the majority of connectionist designs can in theory be made at least as safe as a human,
if you knew exactly what you are doing. The main reason why they're so unsafe is that it's almost impossible for humans to know what they're doing with connectionist designs; there's way too much tangled up complexity to have a hope of untangling it. The other reason is that our design goal for a 'Friendly' AI is actually a rather higher standard than the safety of a human upload. We don't know how likely it is that a human mind would go subtly or overtly mad if it tried to self-improve - but to be honest, it seems quite likely, in absence of a seriously transhuman intelligence on hand to help - and any sensible person would be nervous about trusting one human mind or a small group of human minds with that much potential power anyway.
In your scenario where you do have superintelligence around to help with (or outright do) design and debugging, and the connectionist AGIs don't have the prospect of being much smarter than everything around them, they're relatively safe to experiment with.
your doom and gloom talk about them made them seem very fascinating (see now how this struck back at your intentions)
Yeah, the 'emergence mysticism' people think it's wonderful how arbitrary behavior can arise from messy opaque systems that they build but don't understand. They should stick to having kids, we'd all be a lot better off.
![Smile :)](./images/smilies/icon_smile.gif)
I guess puzzling out arbitrary connectionist intelligences is a nice challenge to keep yourself occupied post Singuarity - if you're a congenital intellectual masochist. Fascinating does not equate to useful though.
Oh and I was wondering about this when I read about your perfect communication is possible for AIs part. What if a human made AI would meet an alien made AI in the future? Would they have instant perfect communication as well, or just a much easier time hashing out a common language?
'Instant perfect communication' works because the AIs already have compatible world models, or because they can write translation layers easily by tracing each other's models up from obvious shared axioms (maths, logic, sensory primitives etc). The later would be a fairly hard challenge for human programmers (much less hard than making AGI), but it should be pretty trivial for AGIs capable of rewriting their own entire code base from first principles (which transhuman AGIs will be). Unsurprisingly, the process is much easier for transparent, rational AGIs than connectionist ones, because you don't have to burn ridiculous amounts of compute power (paying the general connectionist inefficiency penalty to do so) sorting out all the low-level messiness.
In the case of a human AGI meeting an alien one, the common basic layer is lacking; there aren't any agreed comms protocols, programming languages or well-known reference databases. Developing all that from scratch is a task roughly comparable in scope to humans meeting organic aliens and trying to learn each other's language, or develop a common language. However two transhuman intelligences working on the problem at typical AGI clockspeeds may well establish communication so fast that it might as well be instant for human observers. The bandwidth advantage also comes into play, in that an AGI can send massively redundant information and expect its opposite number to consistency-check it, and query any discrepancies. That should eliminate a lot of the potential for misunderstandings that humanlike minds would have, working on a relatively tiny set of utterances (or taking many years to digest dictonaries and encyclopedias).
What if say the alien AI is a hostile one resulting from a connectivist design gone awry and it is hostile to all other intelligences, what if it came to a "fight" with a an AI desgned around a reflective design as opposed to a connectivist one? Does one design trump the other in how effective it can be, or is that kind of thinking not relevant after a certain threshold has been passed?
Note that 'connectionist' is a very broad category. When I talk about 'connectionism', I usually mean the vaguely brainlike 'emergent soups' that are popular with a lot of AGI researchers ATM. However it would be possible to make a connectionist AGI that is both strictly Bayesian and Utilitarian. It would still be rather opaque, inflexible (at the micro scale) and inefficient on anything like current hardware (either CPU-like or FPGA-like). There might conceivably be future hardware on which such a design is a good idea though (say if something like a Trek 'positronic net' could really be built). The transpareny/opacity distinction is critical for us as humans trying to build Friendly AIs, because AGI designs have to be exceptionally 'transparent' for us to have a hope of verifying them. It is much less relevant once the superintelligence transition has passed; 'opacity' just makes reflection more expensive (in computational terms), and that might be a price worth paying on some hardware. However the rational/irrational distinction is
always relevant; probability calculus is
the optimal way to use information, anything else is going to suck.
So really, one does not trump the other, as long as both are basically rational. 'Transparent' designs with a high control flow flexibility, that sacrifice some parallelism and absolute compute power for serial speed, using global or near-global preference functions (and some other highly technical stuff I won't detail here) - all this seems optimal on hardware we can project using current physics and design concepts. But you can always make up for finesse with enough brute force; some amount of computing power, physical resources and/or favorable circumstances will allow the less-optimal (i.e. probably the 'connectionist') AGI to overcome its design inefficiency and win the fight, as long as it is not
pathologically irrational.
There is also the question of why the less efficient AGI hasn't self-modified to resemble the more efficient one, if not before they met, then afterwards once it has a good idea of how the superior design works. There are various possible answers to that, e.g. the less efficient one has a highly distributed and fuzzy goal system which it can't losslessly translate into the new design (either in absolute or due to computational infeasibility). Just make sure you have one.
As a final note, realistically the difference in architecture between two competing transhuman AGIs is unlikely to be a good match with current AI terminology, which is mostly a holdover from 50 years of narrow AI experiments. By that point there are so many layers, concepts, sublties and special cases involved that more likely than not, 'connectionist' vs 'symbolic' won't be useful labels. Don't let that stop you using them in a story though. They're some of the few AI terms that normal readers may actually have heard of, they're being used in a genuine way, and you've already done way more research than most writers bother with, so I'd say go for it.
muon wrote:Since this work is likely to be frequently referenced via general online searches though, I would submit that an even clearer and more direct explanation of the existential risks inherent in the field is well-warranted here.
I am not particularly skilled as a general science writer. For a clear explanation of that issue, I would recommend a paper such as
Artificial Intelligence as a Positive and Negative Factor in Global Risk. I do have a decent amount of practical experience with both commercial narrow AI work and high-level general AI research, and I've spent a lot of time ploughing through the literature and corresponding with other researchers. Thus the aim of this thread; providing answers to specific technical, cultural and historical questions about AI, that people on this forum were curious about. I will try to speculate on futurist things that writers are interested in (e.g. what are societies of AIs like) when asked, but for many of those questions, no one on earth can give a concrete, definite answer.
Especially in light of the fact that many current projects outright refuse to even acknowledge there is a risk! Screaming at those types, "you have insufficient fear!", may be futile, yet some attempt must be made, agreed?
Agreed. Fortunately it's not my problem as such, the SIAI is specifically funded to do that, and I wish them the best of luck with it. I'll give it a shot if I'm in a position to do so, but I've long since stopped making deliberate efforts to convince other AGI projects to take safety more seriously.
In particular, the mention of scenarios where "an AI could escape onto the Internet" imply that containment (or even management!) of an SI after a hard takeoff is theoretically feasible.
It isn't, as discussed in the 'Robots Learn to Lie' thread. You don't even need to have the argument about how easy it is for an AGI to get out of the 'box', or how long you could expect humans to maintain
perfect security even if such security was possible in the first place. In any case, the notion that you can reliable keep a superintelligence in a box is not the most serious mistake made by people in favor of this scenario.
The 'AI box' argument is irrelevant because an AGI in a box is useless, and it would be a waste of effort to develop one if you're never going to give it a significant way of interacting with the world. The key mistake made by proponents of 'AI boxing' is that given an AGI of dubious reliability, you can become highly confident that it is benevolent simply by testing it in simulated scenarios, possibly while watching a debugger trace of its mind state. They imagine that they could keep an AGI in a box while it is 'made safe' (or 'shown to be safe' if they're extremely optimistic and think that AGI is benevolent by default). This is simply not the case even given a 100% reliable box.
Firstly there is absolutely no way human developers are going to make a simulation of reality realistic enough to fool a transhuman intelligence (into revealing its true goals). Secondly general AIs are
extremely hard to analyse (with 'white box' methods) even under the ideal case of a transparent design that is not being actively deceptive. Most AGI designs are in no way transparent and we have to assume active deception is a possibility at all times. Thirdly even if you get lucky and the AGI is genuinely benevolent, there is no guarentee that it will stay that way in the future. For a complex nonlinear system such as a self-modifying rational intelligence (which is incidentally far more nonlinear than a human), past behavior does not guarantee future behavior. The only way to do that is an explicit proof of the stability of the goal system - a complex problem that inevitably extends to a functional verification of the entire design. Having even a reliable 'box' would not help with this at all. As such the usefulness of boxing and black-box testing combined is
solely as a last-ditch backup line of defence, which may save you from mistakes made in the formal friendliness design/proof stage.
Once we launch a hard takeoff, the SI will escape (via indirect memetic transformation of human society if nothing else), and if it's not Friendly, we all go straight to Hell and there's nothing we could do about it.
Strictly, we all go to hell only if the AI has a fascination with running uploads or simulations of past humans, for experimentation or more bizarre reasons. In the normal case, we just die.
(I grant that is far from a rigorously proven statement, normally rejected on SDN, but it would be foolhardy to assume the converse, particularly in light of what we already know about the likely capabilities of even a modestly transhuman AI.)
Yes. When you deliberately play with existential risks, you (should) make the most conservative assumptions still within the bounds of plausibility (frankly you should then add another order of magnitude or two just to account for your likely broken notion of 'plausible', because no one has a complete understanding of these risks yet).