dark mode toggle what even is an institution? 2022 dec 30 see all posts special thanks to dennis pourteaux and tina zhen for discussion that led to this post. a recent alternative political compass proposed by dennis pourteaux proposes that the most important political divide of our present time is not liberty vs authoritarianism or left vs right, but rather how we think about "institutions". are the institutions that society runs on today good or bad, and is the solution to work incrementally to improve them, replace them with radically different institutions, or do away with institutions altogether? this, however, raises a really important question: what even is an "institution" anyway? the word "institution" in political discourse brings to mind things like national governments, the new york times, universities and maybe the local public library. but the word also gets used to describe other kinds of things. the phrase "the institution of marriage" is common in english-language discourse, and gets over two million search results on google. if you ask google point-blank, "is family an institution", it answers yes. chatgpt agrees: if we take chatgpt's definition that "a social institution is a pattern of behaviors and norms that exist within a society and are thought to be essential to its functioning" seriously, then the new york times is not an institution no one argues that it's literally essential, and many people consider it to be actively harmful! and on the other side, we can think of examples of things that maybe are institutions that pourteaux's "anti-institutionalists" would approve of! twitter the bitcoin or ethereum blockchains the english language substack markets standards organizations dealing with international shipping this leads us to two related, but also somewhat separate, questions: what is really the dividing line that makes some things "institutions" in people's eyes and others not? what kind of world do people who consider themselves anti-institutionalists actually want to see? and what should an anti-institutionalist in today's world be doing? a survey experiment over the past week, i made a series of polls on mastodon where i provided many examples of different objects, practices and social structures, and asked: is this an institution or not? in some cases, i made different spins on the same concept to see the effects of changing some specific variables. there were some fascinating results. here are a few examples: and: and: and, of course: there's more fun ones: nyt vs russia today vs bitcoin magazine, the solar system vs what if we started re-engineering it, prediction markets, various social customs, and a lot more. here, we can already start to see some common factors. marriage is more institution-y than romantic relationships, likely because of its official stamp of recognition, and more mainstream relationship styles are more institution-y than less mainstream styles (a pattern that repeats itself when comparing nyt vs russia today vs bitcoin magazine). systems with clearly visible human beings making decisions are more institution-y than more impersonal algorithmic structures, even if their outputs are ultimately entirely a function of human-provided inputs. to try to elucidate things further, i decided to do a more systematic analysis. what are some common factors? robin hanson recently made a post in which he argued that: at least on prestigious topics, most people want relevant institutions to take the following ideal form: masses recognize elites, who oversee experts, who pick details. this seemed to me to be an important and valuable insight, though in a somewhat different direction: yes, that is the style of institution that people find familiar and are not weirded out by (as they might when they see many of the "alternative institutions" that hanson likes to propose), but it's also exactly the style of institutions that anti-institutionalists tend to most strongly rail against! mark zuckerberg's very institution-y oversight board certainly followed the "masses recognize elites who oversee experts" template fairly well, but it did not really make a lot of people happy. i decided to give this theory of institution-ness, along with some other theories, a test. i identified seven properties that seemed to me possible important characteristics of institutions, with the goal of identifying which ones are most strongly correlated to people thinking of something as being an institution: does it have a "masses recognize elites" pattern? does it have a "elites oversee experts" pattern? is it mainstream? is it logically centralized? does it involve interaction between people? (eg. intermittent fasting doesn't, as everyone just chooses whether or not to do it separately, but a government does) does it have a specific structure that has a lot of intentional design behind it? (eg. corporations do, friendship doesn't) does it have roles that take on a life independent of the individuals that fill them? (eg. democratically elected governments do, after all they even call the leader "mr. president", but a podcast which is named after its sole host does not at all) i went through the list and personally graded the 35 maybe-institutions from my polls on these categories. for example, tesla got: 25% on "masses recognize elites" (because it's run by elon musk, who does in practice have a lot of recognition and support as a celebrity, but this isn't a deeply intrinsic feature of tesla, elon won't get kicked out of tesla if he loses legitimacy, etc) 100% on "elites oversee experts" (all large corporations follow this pattern) 75% on "is mainstream" (almost everyone knows about it, lots of people have them, but it's not quite a new york times-level household name) 100% on "logical centralization" (most things get 100% on this score; as a counterexample, "dating sites" get 50% because there are many dating sites and "intermittent fasting" gets 0%) 100% on "involves interaction between people" (tesla produces products that it sells to people, and it hires employees, has investors, etc) 75% on "intentional structure" (tesla definitely has a deep structure with shareholders, directors, management, etc, but that structure isn't really part of its identity in the way that, say, proof of stake consensus is for ethereum or voting and congress are for a government) 50% for "roles independent of individuals" (while roles in companies are generally interchangeable, tesla does get large gains from being part of the elon-verse specifically) the full data is here. i know that many people will have many disagreements over various individual rankings i make, and readers could probably convince me that a few of my scores are wrong; i am mainly hoping that i've included a sufficient number of diverse maybe-instiutions in the list that individual disagreement or errors get roughly averaged out. here's the table of correlations: masses recognize elites 0.491442156943094 elites oversee experts 0.697483431580409 is mainstream 0.477135770662517 logical centralization 0.406758324754985 interaction between people 0.570201749796132 intelligently designed structure 0.365640100778201 roles independent of individuals 0.199412937985826 but as it turns out, the correlations are misleading. "interaction between people" turns out to be an almost unquestionably necessary property for something to have to be an institution. the correlation of 0.57 kind of shows it, but it understates the strength of the relationship: literally every thing that i labeled as clearly involving interaction had a higher percentage of people considering it an institution than every thing i labeled as not involving interaction. the single dot in the center is my hypothetical example of an island where people with odd-numbered birthdays are not allowed to eat meat before 12:00; i didn't want to give it 100% because the not-meat-eating is a private activity, but the question still strongly implies some social or other pressure to follow the rule so it's also not really 0%. this is a place where spearman's coefficient outperforms pearson's, but rather than spurting out exotic numbers i'd rather just show the charts. here are the other six: the most surprising finding for me is that "roles independent of individuals" is by far the weakest correlation. twitter run by a democracy is the most institution-y of all, but twitter run by a pay-to-govern scheme is as institution-y as twitter that's just run by elon directly. roles being independent of individuals adds a guarantee of stability, but roles being independent of individuals in the wrong way feels too unfamiliar, or casual, or otherwise not institution-like. dating sites are more independent of individuals than professional matchmaking agencies, and yet it's the matchmaking agencies that are seen as more institution-like. attempts at highly role-driven and mechanistic credibly-neutral media, (eg. this contraption, which i actually think would be really cool) just feel alien perhaps in a bad way, but also perhaps in a good way, if you find the institutions of today frustrating and you're open-minded about possible alternatives. correlations with "masses recognize elites" and "elites oversee experts" were high; higher for the second than the first, though perhaps hanson and i had different meanings in mind for "recognize". the "intentional structure" chart has an empty bottom-right corner but a full top-left corner, suggesting that intentional structure is necessary but not sufficient for something to be an institution. that said, my main conclusion is probably that the term "institution" is a big mess. rather than the term "institution" referring to a single coherent cluster of concepts (as eg. "high modernism" does), the term seems to have a number of different definitions at play: a structure that fits the familiar pattern of "masses recognize elites who oversee experts" any intentionally designed large-scale structure that mediates human interaction (including things like financial markets, social media platforms and dating sites) widely spread and standardized social customs in general i suspect that anti-institutionalists focus their suspicion on (1), and especially instances of (1) that have been captured by the wrong tribe. whether a structure is personalistic or role-driven does not seem to be very important to anti-institutionalists: both personalities ("klaus schwab") and bureaucracies ("woke academics") are equally capable of coming from the wrong tribe. anti-institutionalists generally do not oppose (3), and indeed in many cases want to see (3) replace (1) as much as possible. support for (2) probably maps closely to pourteaux's "techno-optimist" vs "techno-minimalist" distinction. techno-minimalists don't see things like twitter, substack, bitcoin, ethereum, etc as part of the solution, though there are "bitcoin minimalists" who see the bitcoin blockchain as a narrow exception and otherwise want to see a world where things like family decide more of the outcomes. "techno-optimist anti-institutionalists" are specifically engaged in a political project of either trying to replace (1) with the right kind of (2), or trying to reform (1) by introducing more elements of the right kind of (2). which way forward for anti-institutionalists or institutional reformers? it would be wrong to ascribe too much intentional strategy to anti-institutionalists: anti-institutionalism is a movement that is much more united in what is against than in support of any specific particular alternative. but what is possible is to recognize this pattern, and ask the question of which paths forward make sense for anti-institutionalists. from a language point of view, even using the word "institution" at all seems more likely to confuse than enlighten at this point. there is a crucial difference between (i) a desire to replace structures that contain enshrined elite roles with structures that don't, (ii) a preference for small-scale and informal structures over large-scale and formal ones, (iii) a desire to simply swap the current elites out for new elites, and (iv) a kind of social libertinist position that individuals should be driven by their own whims and not by incentives created by other people. the word "institution" obscures that divide, and probably focuses too much attention on what is being torn down rather than what is to be built up in its place. different anti-institutionalists have different goals in mind. sure, the person on twitter delivering that powerful incisive criticism of the new york times agrees with you on how society should not be run, but are you sure they'll be your ally when it comes time to decide how society should be run? the challenge with avoiding structures entirely is clear: prisoner's dilemmas exist and we need incentives. the challenge with small-scale and informal structures is often clear: economies of scale and gains from standardization though sometimes there are other benefits from informal approaches that are worth losing those gains. the challenge with simply swapping the elites is clear: it has no path to socially scale into a cross-tribal consensus. if the goal is not to enshrine a new set of elites forever, but for elites to permanently be high-churn (cf. balaji's founder vs inheritor dichotomy), that is more credibly neutral, but then it starts getting closer to the territory of avoiding enshrined elites in general. creating formal structures without enshrined elites is fascinating, not least because it's under-explored: there's a strong case that institutions with enshrined elite roles might be an unfortunate historical necessity from when communication was more constrained, but modern information technology (including the internet and also newer spookier stuff like zero-knowledge cryptography, blockchains and daos) could rapidly expand our available options. that said, as hanson points out, this path has its own fair share of challenges too. dark mode toggle different types of layer 2s 2023 oct 31 see all posts special thanks to karl floersch for feedback and review the ethereum layer 2 ecosystem has been expanding rapidly over the last year. the evm rollup ecosystem, traditionally featuring arbitrum, optimism and scroll, and more recently kakarot and taiko, has been progressing quickly, making great strides on improving their security; the l2beat page does a good job of summarizing the state of each project. additionally, we have seen teams building sidechains also starting to build rollups (polygon), layer 1 projects seeking to move toward being validiums (celo), and totally new efforts (linea, zeth...). finally, there is the not-just-evm ecosystem: "almost-evms" like zksync, extensions like arbitrum stylus, and broader efforts like the starknet ecosystem, fuel and others. one of the inevitable consequences of this is that we are seeing a trend of layer 2 projects becoming more heterogeneous. i expect this trend to continue, for a few key reasons: some projects that are currently independent layer 1s are seeking to come closer to the ethereum ecosystem, and possibly become layer 2s. these projects will likely want a step-by-step transition. transitioning all at once now would cause a decrease in usability, as the technology is not yet ready to put everything on a rollup. transitioning all at once later risks sacrificing momentum and being too late to be meaningful. some centralized projects want to give their users more security assurances, and are exploring blockchain-based routes for doing so. in many cases, these are the projects that would have explored "permissioned consortium chains" in a previous era. realistically, they probably only need a "halfway-house" level of decentralization. additionally, their often very high level of throughput makes them unsuitable even for rollups, at least in the short term. non-financial applications, like games or social media, want to be decentralized but need only a halfway-house level of security. in the social media case, this realistically involves treating different parts of the app differently: rare and high-value activity like username registration and account recovery should be done on a rollup, but frequent and low-value activity like posts and votes need less security. if a chain failure causes your post to disappear, that's an acceptable cost. if a chain failure causes you to lose your account, that is a much bigger problem. a big theme is that while applications and users that are on the ethereum layer 1 today will be fine paying smaller but still visible rollup fees in the short term, users from the non-blockchain world will not: it's easier to justify paying $0.10 if you were paying $1 before than if you were paying $0 before. this applies both to applications that are centralized today, and to smaller layer 1s, which do typically have very low fees while their userbase remains small. a natural question that emerges is: which of these complicated tradeoffs between rollups, validiums and other systems makes sense for a given application? rollups vs validiums vs disconnected systems the first dimension of security vs scale that we will explore can be described as follows: if you have an asset that is issued on l1, then deposited into the l2, then transferred to you, what level of guarantee do you have that you will be able to take the asset back to the l1? there is also a parallel question: what is the technology choice that is resulting in that level of guarantee, and what are the tradeoffs of that technology choice? we can describe this simply using a chart: system type technology properties security guarantees costs rollup computation proven via fraud proofs or zk-snarks, data stored on l1 you can always bring the asset back to l1 l1 data availability + snark-proving or redundant execution to catch errors validium computation proven via zk-snarks (can't use fraud proofs), data stored on a server or other separate system data availability failure can cause assets to be lost, but not stolen snark-proving disconnected a separate chain (or server) trust one or a small group of people not to steal your funds or lose the keys very cheap it's worth mentioning that this is a simplified schema, and there are lots of intermediate options. for example: between rollup and validium: a validium where anyone could make an on-chain payment to cover the cost of transaction fees, at which point the operator would be forced to provide some data onto the chain or else lose a deposit. between plasma and validium: a plasma system offers rollup-like security guarantees with off-chain data availability, but it supports only a limited number of applications. a system could offer a full evm, and offer plasma-level guarantees to users that do not use those more complicated applications, and validium-level guarantees to users that do. these intermediate options can be viewed as being on a spectrum between a rollup and a validium. but what motivates applications to choose a particular point on that spectrum, and not some point further left or further right? here, there are two major factors: the cost of ethereum's native data availability, which will decrease over time as technology improves. ethereum's next hard fork, dencun, introduces eip-4844 (aka "proto-danksharding"), which provides ~32 kb/sec of onchain data availability. over the next few years, this is expected to increase in stages as full danksharding is rolled out, eventually targeting around ~1.3 mb/sec of data availability. at the same time, improvements in data compression will let us do more with the same amount of data. the application's own needs: how much would users suffer from high fees, versus from something in the application going wrong? financial applications would lose more from application failures; games and social media involve lots of activity per user, and relatively low-value activity, so a different security tradeoff makes sense for them. approximately, this tradeoff looks something like this: another type of partial guarantee worth mentioning is pre-confirmations. pre-confirmations are messages signed by some set of participants in a rollup or validium that say "we attest that these transactions are included in this order, and the post-state root is this". these participants may well sign a pre-confirmation that does not match some later reality, but if they do, a deposit gets burned. this is useful for low-value applications like consumer payments, while higher-value applications like multimillion-dollar financial transfers will likely wait for a "regular" confirmation backed by the system's full security. pre-confirmations can be viewed as another example of a hybrid system, similar to the "plasma / validium hybrid" mentioned above, but this time hybridizing between a rollup (or validium) that has full security but high latency, and a system with a much lower security level that has low latency. applications that need lower latency get lower security, but can live in the same ecosystem as applications that are okay with higher latency in exchange for maximum security. trustlessly reading ethereum another less-thought-about, but still highly important, form of connection has to do with a system's ability to read the ethereum blockchain. particularly, this includes being able to revert if ethereum reverts. to see why this is valuable, consider the following situation: suppose that, as shown in the diagram, the ethereum chain reverts. this could be a temporary hiccup within an epoch, while the chain has not finalized, or it could be an inactivity leak period where the chain is not finalizing for an extended duration because too many validators are offline. the worst-case scenario that can arise from this is as follows. suppose that the first block from the top chain reads some data from the leftmost block on the ethereum chain. for example, someone on ethereum deposits 100 eth into the top chain. then, ethereum reverts. however, the top chain does not revert. as a result, future blocks of the top chain correctly follow new blocks from the newly correct ethereum chain, but the consequences of the now-erroneous older link (namely, the 100 eth deposit) are still part of the top chain. this exploit could allow printing money, turning the bridged eth on the top chain into a fractional reserve. there are two ways to solve this problem: the top chain could only read finalized blocks of ethereum, so it would never need to revert. the top chain could revert if ethereum reverts. both prevent this issue. the former is easier to implement, but may cause a loss of functionality for an extended duration if ethereum enters an inactivity leak period. the latter is harder to implement, but ensures the best possible functionality at all times. note that (1) does have one edge case. if a 51% attack on ethereum creates two new incompatible blocks that both appear finalized at the same time, then the top chain may well lock on to the wrong one (ie. the one that ethereum social consensus does not eventually favor), and would have to revert to switch to the right one. arguably, there is no need to write code to handle this case ahead of time; it could simply be handled by hard-forking the top chain. the ability of a chain to trustlessly read ethereum is valuable for two reasons: it reduces security issues involved in bridging tokens issued on ethereum (or other l2s) to that chain it allows account abstraction wallets that use the shared keystore architecture to hold assets on that chain securely. is important, though arguably this need is already widely recognized. (2) is important too, because it means that you can have a wallet that allows easy key changes and that holds assets across a large number of different chains. does having a bridge make you a validium? suppose that the top chain starts out as a separate chain, and then someone puts onto ethereum a bridge contract. a bridge contract is simply a contract that accepts block headers of the top chain, verifies that any header submitted to it comes with a valid certificate showing that it was accepted by the top chain's consensus, and adds that header to a list. applications can build on top of this to implement functionality such as depositing and withdrawing tokens. once such a bridge is in place, does that provide any of the asset security guarantees we mentioned earlier? so far, not yet! for two reasons: we're validating that the blocks were signed, but not that the state transitions are correct. hence, if you have an asset issued on ethereum deposited to the top chain, and the top chain's validators go rogue, they can sign an invalid state transition that steals those assets. the top chain still has no way to read ethereum. hence, you can't even deposit ethereum-native assets onto the top chain without relying on some other (possibly insecure) third-party bridge. now, let's make the bridge a validating bridge: it checks not just consensus, but also a zk-snark proving that the state of any new block was computed correctly. once this is done, the top chain's validators can no longer steal your funds. they can publish a block with unavailable data, preventing everyone from withdrawing, but they cannot steal (except by trying to extract a ransom for users in exchange for revealing the data that allows them to withdraw). this is the same security model as a validium. however, we still have not solved the second problem: the top chain cannot read ethereum. to do that, we need to do one of two things: put a bridge contract validating finalized ethereum blocks inside the top chain. have each block in the top chain contain a hash of a recent ethereum block, and have a fork choice rule that enforces the hash linkings. that is, a top chain block that links to an ethereum block that is not in the canonical chain is itself non-canonical, and if a top chain block links to an ethereum block that was at first canonical, but then becomes non-canonical, the top chain block must also become non-canonical. the purple links can be either hash links or a bridge contract that verifies ethereum's consensus. is this enough? as it turns out, still no, because of a few small edge cases: what happens if ethereum gets 51% attacked? how do you handle ethereum hard fork upgrades? how do you handle hard fork upgrades of your chain? a 51% attack on ethereum would have similar consequences to a 51% attack on the top chain, but in reverse. a hard fork of ethereum risks making the bridge of ethereum inside the top chain no longer valid. a social commitment to revert if ethereum reverts a finalized block, and to hard-fork if ethereum hard-forks, is the cleanest way to resolve this. such a commitment may well never need to be actually executed on: you could have a governance gadget on the top chain activate if it sees proof of a possible attack or hard fork, and only hard-fork the top chain if the governance gadget fails. the only viable answer to (3) is, unfortunately, to have some form of governance gadget on ethereum that can make the bridge contract on ethereum aware of hard-fork upgrades of the top chain. summary: two-way validating bridges are almost enough to make a chain a validium. the main remaining ingredient is a social commitment that if something exceptional happens in ethereum that makes the bridge no longer work, the other chain will hard-fork in response. conclusions there are two key dimensions to "connectedness to ethereum": security of withdrawing to ethereum security of reading ethereum these are both important, and have different considerations. there is a spectrum in both cases: notice that both dimensions each have two distinct ways of measuring them (so really there's four dimensions?): security of withdrawing can be measured by (i) security level, and (ii) what percent of users or use cases benefit from the highest security level, and security of reading can be measured by (i) how quickly the chain can read ethereum's blocks, particularly finalized blocks vs any blocks, and (ii) the strength of the chain's social commitment to handle edge cases such as 51% attacks and hard forks. there is value in projects in many regions of this design space. for some applications, high security and tight connectedness are important. for others, something looser is acceptable in exhcnage for greater scalability. in many cases, starting with something looser today, and moving to a tighter coupling over the next decade as technology improves, may well be optimal. dark mode toggle 預測市場：一個選舉小故事（2021年 二月） 2020 feb 18 see all posts 感谢jerry ho翻译这个文章 特別感謝 jeff coleman, karl floersch, robin hanson 協助校稿並給了許多重要的回饋。 不適警告：我在文中表示了自己的部分政治立場。 預測市場是我多年來一直感興趣的主題。讓大眾對未來的事件進行投注，然後把這些投注的賠率當作預測值，得到可信且中立的「預測事件發生之機率」。這種機制設計帶來的應用讓我覺得非常有趣。類似的想法像是futarchy，一直都很讓我感興趣，因為它們是可以改善治理和決策的創新工具。而正如augur和omen，還有最近的polymarket，預測市場也是區塊鏈（在上述三種應用下，ethereum）的一個吸引人的應用。 在2020年美國總統大選，預測市場似乎終於進入了鎂光燈下，像是區塊鏈上的市場，就從2016年的近乎不存在，到2020年擁有了數以百萬美元的交易量。我個人密切關注ethereum應用跨越鴻溝被大眾採用，所以這當然引起了我的興趣。起初我傾向於只看，自己不參與：我不是美國選舉政治的專家，我為什麽會自以為我的觀點會比已經在交易的人更正確呢？但，在我的twitter同溫層中，我看到了越來越多我尊敬的「非常聰明的人」，提出了市場事實上真的很不理性的論點，所以如果可以的話我應該參與市場然後和他們對賭。最後我被說服了。 我決定在這個我參與創立的區塊鏈上做個實驗：我在augur買了價值$2000的ntrump（如果川普輸了，每個token值$1）。我當時太天真，沒有想到我後來的部位會增加到$308,249，賺了超過$56,803的利潤。更沒想到的是，我還會把剩下的賭注全部拿來和願意和我交易的對手對賭，而且還是下在川普已經輸掉大選之後。在接下來的兩個月裡發生的事情，會成為社會心理學、專業知識、套利和市場效率極限的一個產生十分重要結果的案例研究。這樣的結果對任何對想要設計經濟誘因機制的人有著重要的意義。 選前 我在這次大選的第一個賭注其實根本不是押在區塊鏈上。當kanye在7月宣布競選總統時，有一位我讓我相當尊重的政治學家（因為他的思路獨特且高水準），在twitter上聲稱他有信心這會分化反川普的選票，導致川普獲勝。我記得，當時我認為他這種特別的觀點有些過於自信，甚至有可能是在腦中過度推論造成的結果 如果一個觀點看起來很睿智又很反直覺，那這個觀點一定是對的。所以我當然提出了$200的賭注，賭的是無聊的那種支持拜登的常見觀點。他與有榮焉的接受了。 9月，大選話題又出現在我的視野內，這次是預測市場引起了我的注意。市場給出了川普近50%的勝算。但在我的twitter同溫層中，我看到了越來越多我尊敬的「非常聰明的人」，都在說著一件事：這個數字似乎太高了。這當然就引出了我們熟悉的「效率市場之爭」：如果你能以$0.52的價格買到一個如果川普輸了就給你$1的token，而川普實際輸掉的機率比這要高得多，為什麽人們不乾脆來買這個token，這樣價格最後才會漲得更高？如果沒有人這麽做，你又憑什麽認為自己比別人聰明？ ne0liberal在選舉日之前的推很好地總結了他當時覺得預測市場不準確的理由。簡單來說，2020年以前大多數人使用的（非區塊鏈）預測市場有各種限制，使得人們只能用少量的現金來參與。因此，如果一個非常聰明的個人或專業機構看到了一個他們認為是錯誤的機率，他們把價格推向「他們認為正確的方向」的能力相當受限。 文章中指出的重要限制包含: * 每人可投注的金額很低（遠低於$1000） * 高額的手續費（如predictit的5%提款費用） 而這也是我在9月份反駁ne0liberal的原因：雖然呆板的舊世界中心化預測市場有很高的手續費和很低的投注上限，但加密貨幣市場沒有！在augur或omen上，如果有人認為某個預測結果token的價格過低或過高，那麽他可以無限制的買入或賣出。而當時基於區塊鏈的預測市場趨勢和predictit的相仿。如果真的是因為高手續費和低投注上限的關係，讓頭腦更冷靜的交易者無法戰勝過於樂觀的交易者，才導致川普最後被高估的話，那麽為什麽明明區塊鏈市場沒有這些問題，卻也有著同樣的價格走勢呢？ 我twitter上的朋友對此給出的主要回應是，區塊鏈預測市場極端小眾，只有很少人在用，而要在很少人中找到很瞭解政治的人就更少了，所以他們沒辦法輕易接觸到加密貨幣。這看起來似乎很有道理，但我對這個論點不太有信心。於是我押了$2000對賭川普當選，就沒有再繼續下去。 大選中 然後，選舉發生了。一開始雖然我們以為自己預測錯誤，被川普贏得的大量席位嚇死，拜登還是最終的贏家。至於究竟大選本身到底有沒有吻合，還是不吻合預測市場的效率嘛，據我所知，這是一個大哉問。一方面，通過標準的貝氏定理，我應該降低對預測市場的信心，至少我應該更相信nate silver。預測市場給出了拜登60%的勝算，而nate silver給出了拜登90%的勝算。既然拜登事實上贏了，這就是一個證據，證明在我生活的世界中nate的答案比較正確。 但另一方面，你可以說預測市場對贏多少的估計更勝一籌。nate機率分布的中位數是538張選舉人票中的370張左右會給拜登： 雖然川普市場沒有給出機率分布，但如果你要從「川普有40%的勝算」這個統計中推一個機率分布，你的中位數可能會是拜登拿到300張選票人票左右。實際結果是：306票。所以，預測市場與nate的孰優孰劣，在我看來，還在未定之天。 選後 但當時我無法想像的是，選舉本身只是一個開始。選舉結束幾天後，拜登就被各大組織，甚至一些外國政府宣布當選。川普不出意料的對選舉結果發起了各種法律戰，但這些舉動很快就失敗了。儘管如此，在一個多月的時間裡，ntrump token的價格一直保持在85美分！ 一開始，人們似乎有理由猜測川普有15%的機會推翻選舉結果，畢竟他任命了三名法官進入最高法院，當黨派紛爭加劇的時候，許多法官都開始站隊而非堅守原則。在接下來的三周裡，明顯看得出來這些法律戰都失敗了，雖然川普的希望也隨著時間的推移繼續顯得更加嚴峻，但ntrump的價格並沒有動搖；事實上，它甚至短暫地下降到$0.82左右。在12月11日，大選結束已經過了5個星期的時候，最高法院果斷地一致駁回了川普推翻投票結果的企圖，這讓ntrump價格終於漲到了$0.88。 到了11月，我終於相信市場懷疑論者是對的，所以我自己也投身其中，與川普對賭。這個決定並不是為了錢，畢竟在不到兩個月之後，光是我手上有的dogecoin升值就能讓我賺得比預測市場更多，讓我能夠捐給givedirectly。所以不如說這是為了參與一場實驗，不僅是作為一個觀察者，而是作為一個積極的參與者，這樣能提高我對其他人的理解，到底為什麼大家沒有趕在我之前蜂擁而上購買ntrump。 進場 我的ntrump是在catnip上買的，catnip是一個前端界面，它將augur預測市場和balancer結合在一起，balancer是一個uniswap風格的恒定功能做市商。catnip是迄今為止做預測市場交易最好用的界面，我覺得它對augur的可用性貢獻很大。 有兩種方式可以用catnip對賭川普。 使用dai直接在catnip上購買ntrump。 使用foundry呼叫augur功能，將1個dai轉換成1個ntrump+1個ytump+1個itrump（「i」代表「無效」，後面會詳細介紹），然後將ytrump賣到catnip上 一開始，我只知道第一個選項。但後來我發現balancer上ytrump的流動性更高，所以我轉到第二個選項。 還有一個問題：我沒有任何dai。我有eth，我可以賣出我的eth來獲得dai，但我不想犧牲我的eth曝險部位；就算我在對川普的押注中賺取了$50,000，如果我因為eth價格變化而讓我賠了了$500,000那就得不償失了。所以我決定在makerdao上開立抵押債務頭寸（cdp，現在也叫 "vault"）來保持我的eth曝險價格。 cdp是dai的產生方式：使用者將eth存入一個smart contract，然後可以提出相當於eth價值2/3的dai，這些dai是新產生出來的。他們可以放入與自己之前提出相同金額的dai加上額外的利息費用（目前為3.5%），來取回抵押於其中的eth。如果你存入的eth抵押品價值下降到低於你取出的dai價值的150%，任何人都可以開始清算你的vault，強迫賣掉eth來買回dai，並向你收取高額罰金。因此，在價格突然變動的情況下，最好有較高的抵押率；我用價值超過$3的eth來當我提出$1的cdp抵押品。 回顧一下，下圖顯示了操作過程： 我重複操作了很多次；catnip上的滑價，代表我通常一次只能做大約$5000到$10000的交易，不然價格會變得太不利（如果我跳過foundry，直接用dai購買ntrump，基本上只能交易到接近$1000）。兩個月後，我已經累積了367,000個ntrump。 為什麼其他人不這麼做？ 在我開始解釋之前，我有四個主要假設，想辦法解釋為什麽沒什麼人以85美分的價格買入明明就值$1的token： 擔心augur的smart contract會出事 或者川普的支持者會操縱預言機（一種去中心化的機制讓augur token的持有者可以把token壓在事件結果上，投票決定事件結果），使其回報的結果與發生的事實不符。 資本成本：購買這些token，你必須將資金鎖定兩個月以上，這就讓你不能在期間內使用這些錢，更讓你無法進行其他更好賺的交易。 技術上太複雜了，讓人無法輕易交易。 我想錯了，就算機會天降在你面前，有足夠動力嘗試這種奇怪機會的人真的太少了。 四者聽來都很合理。smart contract出事的話是真 的 危 險，而augur的預言機之前遇到的環境都沒有如此有爭議，缺乏測試。資本成本是真的，而且雖然在預測市場中對賭比較簡單，比在股票市場中更簡單，畢竟你知道預測市場的價格永遠不會超過$1，但鎖在其中的資本還是會讓你時時想到加密貨幣市場中其他同樣有利可圖的選項。在dapp中做交易在技術上畢竟很複雜，所以對未知有一定程度的恐懼很合理。 但在我實際進入金融壕溝戰之後，觀察這個市場的價格演變，讓我對上述的假設有了更多的了解。 對smart contract漏洞的恐懼 起初，我認為「對smart contract漏洞的恐懼」可能佔了很重要的一部分。但隨著時間的推移，我更加相信這大概不是一個主要因素。要想知道為什麽我這樣想，可以比較ytrump和itrump的價格。itrump代表「無效川普」；「無效」是指在一些特殊情況下會觸發的事件結果：當事件描述模糊不清時，當市場結束時真實生活的事件還沒有結束時，當市場本身不道德時（例如暗殺市場），以及其他一些類似情況。在這個市場上，itrump的價格始終保持在$0.02以下。如果有人想通過攻擊市場來賺取利潤，那麽他們不應該$0.15的價格買入ytrump，而應該以$0.02的價格買入itrump會更加有利可圖。如果他們買入大量的itrump，且迫使「無效」的結果被觸發，他們就能賺取50倍的回報。所以如果你害怕被攻擊，買入itrump是目前最理性的反應。然而很少有人這麽做。 另一個反對「smart contract出事很可怕」的論點是，在預測市場以外的每一個加密應用中（例如compound，各種yield farming等），人們對smart contract風險的態度超級樂觀。如果人們只是為了僅僅5-8%的年報率，就願意把錢投入到各種高風險、未經測試的專案中，那麽他們沒有理由在預測市場這邊突然變得這麼謹慎啊？ 資本成本 資本成本 鎖定大量資金所帶來的不便和付出機會成本 跟以前比起來我現在更能體會這件事。僅從augur方面來看，我需要鎖定308,249個dai，平均約兩個月才能獲利$56,803。這樣算下來，年化利率約為175%；這樣看下來，與2020年夏天的各種yield farming熱潮相比，這是相當不錯的選擇。但當你考慮到我必須在makerdao上做的操作時，看起來就沒這麼好賺了。因為我想保持在eth上的曝險不變，所以需要用cdp來獲得dai，而要安全使用cdp需要3倍以上的抵押比例。因此我實際需要鎖定的資金總額在$100萬左右。 現在，利率看起來不那麽好了。如果再加上smart contract被駭客攻擊的可能性（不管有多小），或者史無前例的政治事件真的發生了，這樣的結果看起來還是不太好。 即使如此，假設鎖倉抵押有3倍，augur出事的可能性也有3%（我買itrump是為了以防萬一事件結果變成「無效」，所以我只需要擔心事件結果不小心變成「當選」的風險，或者資金被直接偷走），計算出來之後的無風險利率約為35%，如果你考慮人類直覺對風險的看法，結果上會讓利率看起來更低。當然這個交易還是非常有吸引力，只是換句話說，現在看來這個數字不吸引人也可以理解了，因為那些幣圈的人比較習慣頻繁的100倍漲跌。 川普的支持者，當然不用管這些挑戰：他們只投了$60,000，就能對消掉我$308,249的賭注（因為手續費的關係，我賺得比這個數字少）。當機率接近於0或1時，這個賭局對那些想要把機率推離兩端極值的人非常有利。這不僅解釋了川普的狀況，也解釋了那些小眾但沒有勝算的候選人經常獲得高達5%獲勝機率的原因。 技術難度 我最初嘗試在augur上購買ntrump，但技術問題使我無法直接在augur上下單（我問過的其他人沒有這個狀況...我現在還是不知道這是啥問題) catnip的界面要簡單得多，而且很有效。然而像balancer（和uniswap）這樣的自動做市商對於只在小額交易上有優勢；要做大交易的時候滑點太多。這是「amm vs order book」爭論的一個好例子。amms更方便，但order book在大額交易上確實更好。uniswap v3正在引入有更好資本效率的amm設計；我們可以期待觀察情況是否有改善。 雖然還有其他複雜的技術問題，但這些問題看似都很容易解決。像catnip這樣的界面應該能夠很簡單的把「dai->foundry->賣出ytrump」的路徑整合到合約中，這樣你就可以在一次交易中用這種方式買入ntrump。事實上，這個界面甚至可以檢查「dai -> ntrump」路徑和「dai -> foundry -> 賣出 ytrump」路徑的價格和流動性，然後自動得出更好地交易方式。我們甚至可以把從makerdao cdp中提出dai這個步驟加入其中。我在這里的結論很樂觀：技術複雜度是這次參與市場投注最真實的障礙，但隨著技術的進步，未來幾次將變得更加容易。 聰明而過度缺乏自信 現在我們討論最後一種可能：許多人（尤其是聰明人）都有一種毛病，那就是他們過度謙虛了，他們太容易得出結論：「如果別人沒有採取行動，那一定代表有理由在，所以那個行動其實沒有價值」。 eliezer yudkowsky在他的優秀著作《inadequate equilibria》的後半部分就提出了這個理由，他認為太多人過度採取了「modest epistemology」此種態度（譯註：指心胸過度開放，接受一切不可解釋的現象起源於自己的無知），就算得出的結果代表大多數人其實不理性、懶惰、甚至大多數人只是搞錯了，我們還是應該根據我們自己得出的結果，更大膽的採取行動。當我第一次讀到這些段落的時候，我覺得elizer說錯了，他看起來太過自大。但經歷了這段事情後，我看到了他的立場其實有著一些智慧。 這不是我第一次親身體驗到「相信自己的推理」是多麼好的美德。當我最初開始研究ethereum的時候，我一開始被恐懼困擾，認為這個項目一定有一些不可避免的理由會失敗。你看，一個完全可程式化可以寫smart contract功能的區塊鏈，比之前的區塊鏈好上這麼多，應該很多人都已經趕在我之前想過了對吧。所以我完全覺得我當時一發出這樣的想法後，很多聰明的密碼學家會跑來告訴我ethereum這樣的東西是理論上不可能實現的。然而，從來沒有人這麽做。 當然，並不是每個人都患有過度謙虛的毛病。很多人預測川普會贏得大選，他們明顯被自己過度的為反而反給蒙蔽了。我年輕時努力讓自己不要這麼謙虛且恐懼，ethereum終得以受益，但還有其他更多的項目可以不需要如此的聰明過度而缺乏自信，避免失敗。 此人未為過度謙虛而感到困擾 但在我看來，正如yeats的名言，「最好的人缺乏信念，而最壞的人充滿激情」，這件事現在我看起來是如此的真實。雖然過度自信或為反而反兩者有時候都是錯的，但在我看來，我們應該要這樣想：「全盤相信學術機構、媒體、政府、市場上現存的解方和現象」這種觀點是不對的。仔細想想，這些機構會存在的原因其實正好就是因為 有人覺得其他的機構講的不對，或是他們至少偶爾會出錯 所以才有新的機構與觀點出現。 futarchy給我們上的一課 親眼看到資本成本的重要性及其與風險的相互作用，也顯示了futarchy這種決策系統的重要性。futarchy，以及「決策市場」，可以成為預測市場的一個充滿潛力，對我們的社會非常有用的應用。預測下一任總統是誰就算稍微準一點也沒有多少社會價值，但有條件的預測卻有很大的社會價值：如果我們做了a，達成好結果x的機率有多少，如果我們改做b，那機率又有多少？條件預測之所以重要，是因為它不僅能滿足我們的好奇心，還能幫助我們做出決策。 雖然選舉預測市場的作用遠不如條件預測，但它們可以幫助我們了解一個重要的問題：操縱市場、或是偏見和錯誤意見會給準確率帶來多大影響？我們可以通過觀察套利的難度來回答這個問題：假設條件預測市場目前給出的機率（在你看來）是錯誤的（可能是由於交易者訊息不對稱造成的，或是有人企圖操縱市場；在此原因不重要）。你能在這樣的市場中產生多大的影響，把機率和賠率回正能讓你賺多少錢？ 讓我們從一個具體的例子開始。假設我們正試圖利用預測市場在決策a和決策b之間做出選擇，其中每個決策都有一定機率實現某種理想的結果。假設你的看法是，決策a有50%的機會實現目標，而決策b有45%的機會。與此同時，市場（在你看來是錯誤的）認為決策b有55%的機會，決策a有40%的機會。 認真決策後得到好結果的機會... 現在市場的位置 你的看法 a 40% 50% b 55% 45% 假設你是一個小參與者，所以你的個人投注不會影響結果，只有許多投注者一起行動才會影響結果。你應該下注多少錢？ 這里的標準理論仰賴凱利公式。就是說，無論何時你都應該讓你的資產預期的對數最大化。在這種情況下，我們可以解下面的等式。假設你投資部分的資金r用$0.4元買了a-token。從你的角度來看，你預期的新的對數財富將是： 0.5 * log((1-r) + \frac{r}{0.4}) + 0.5 * log(1-r) 這裡的第一項代表有50%的機會（從你的角度來看）賭注可以有回報，你的投資增長了2.5倍（因為你用$0.4買了$1）。 第二項代表你有50%的機會賭注沒有得到回報，你失去了你所投注的部分。我們可以用微積分來找到正確的r使上式最大化；懶惰的人可以看wolframalpha。答案是r = 1/6。如果其他人買入，市場上a的價格上升到47%（而b的價格下降到48%），我們可以對最後一個會讓市場反轉的交易重新進行計算，這樣就能正確的表示對a有利： 0.5 * log((1-r) + \frac{r}{0.47}) + 0.5 * log(1-r) 我們可以看到預期的對數財富最大化的r值僅為0.0566。結論很清楚：當決策很接近，而且雜訊很大的時候，事實證明投資在市場的資金只能是一小部分，這樣才合理。這已經是假設理性的情況下了，因為大多數人對這種不確定結果的賭博的投資，比凱利公式預言的的要少。再加上資本成本，差距就更大了。但是，如果攻擊者真的不管為了什麼個人理由想要讓結果b發生，他們可以很簡單地將所有資本投進去買該token。總而言之，這個場景對攻擊者相當有利，超過20:1。 當然，現實中的攻擊者很少願意把所有資金押在一個決策上。且futarchy也不是唯一容易被攻擊的機制：股票市場也同樣脆弱，非市場決策機制的部分也可以被有錢又有心的攻擊者以各種方式操縱。盡管如此，我們還是應該特別注意看看，到底futarchy能不能讓我們的決策準確率再創新高。 有趣的是，數學告訴了我們當操縱者想要將結果推向極值時，futarchy反而能發揮最大作用。一個例子是責任保險，像是有人希望以不正當的方式獲得保險理賠時，這相當於強迫把市場估計不利事件發生的機率降到零。事實證明，責任保險是提出futarchy的robin hanson的政策處方簽新寵。 預測市場能變得更好嗎？ 我們最後要問的問題是：預測市場會重蹈覆轍嗎？像是12月初的時候，市場預測了川普有15%的機會推翻選舉結果，甚至在最高法院（包括他任命的三名法官）叫他滾蛋之後，市場還是預測有12%的機會川普能推翻選舉結果。或者隨著時間經過，市場能越變越好？出人意料地，我的答案完全偏樂觀，而且我有幾個保持樂觀的理由。 自然選擇下的市場 首先，這些事件讓我對市場效率和理性究竟有了新的看法。很多時候，市場效率的支持者聲稱，市場之所以會有效率，是因為大多數參與者都是理性的（或者至少理性的人超過了其他愚昧群體），當然這件事是正確的，是個公理(axoim)。所以，讓我們試著從進化中的角度來看待正在發生的事情。 加密貨幣是一個年輕的生態系統，與主流社會還相當脫節。雖然elon最近在twitter上發表了不少言論，我們還是可以看的出來圈子裡的人們對選舉政治學的方方面面不夠瞭解。那些選舉政治學方面的專家很難進入加密貨幣領域，而且加密貨幣圈子有很多為反而反的政治言論都不怎麼正確。但，今年在加密貨幣圈子內發生的事情是這樣的，正確的預期了拜登獲勝的預測市場使用者，其資本得到了18%的增長，而錯誤預測川普獲勝的預測市場使用者的資本得到了100%的減少（他們投入市場的部分）。 因此，選擇壓力會有利於下注結果正確的人。經過十輪之後，好的預測者會有更多的資本去下注，而壞的預測者會有更少的資本去下注。這不需要任何人在過程中「變得更聰明」或「吸取教訓」，更不需要假設其他人類的推理和學習能力。這僅僅是選擇動力學的結果，隨著時間的推移，善於做出正確猜測的參與者將在生態系統中占據主導地位。 請注意，預測市場在這方面比股票市場表現得更好：股票市場的 「新貴」往往是來自於某次幸運的千倍收益，這成為了信號中的雜訊，但在預測市場中，價格被限定在0和1之間，限制了任何一個單一事件能帶來的影響。 參與者更好 技術也更好 第二，預測市場本身會有進步。使用者界面已經有了很大的改進，並將繼續進一步完善。makerdao->foundry->catnip現在雖然很複雜，但會被被抽象為單一交易。區塊鏈的擴容技術也會進步，降低參與者的手續費（內建amm的zk-rollup loopring已經在ethereum主網上線，理論上可以在上面跑一個預測市場）。 第三，我們看到了預測市場正常運行，這樣的例子會讓參與者比較不害怕。使用者會看到，就算在非常有爭議的情況下augur預言機還是能給出正確的結果（這次有兩輪爭議，但no的那一方還是乾淨利落的贏了）。來自加密貨幣領域之外的人，看到這樣的成功的例子，也會更願意參與其中。也許連nate silver自己都可能弄來一些dai，並在2022年以後使用augur、omen、polymarket和其他市場來增加自己的收入。 第四，預測市場技術本身可以改進。這是我自己提出的一個市場設計建議，可以在保持高資本效率的同時，對許多低機率事件下注。這樣能夠防止許多那些低機率事件的賠率偏高。我相信會有其他的想法出現，且我期待看到更多這方面的實驗。 結論 這個預測市場的事件事件始末，身為第一手試驗非常的有趣，可以讓人看著事件在複雜的個人、社會心理學中演化與碰撞。它展示了實際上市場效率的運作方式，市場效率的局限，以及可以改善的方式。 它也很好地展示了區塊鏈的力量；事實上，對我而言它是ethereum應用中最有具體價值的一個。區塊鏈經常被批評為投機玩具，只會自我套娃大玩金融遊戲（token, 放去yield farming, 得到的回報是...其他新出現的token），沒有任何有意義的產出。當然也有批評者沒有注意到的例子；我個人從ens中受益，而且遇過幾次信用卡完全不能用的例子，最後只好使用eth進行支付時。但，在過去的幾個月裡，我們似乎看到了ethereum的應用真的對人們有用了，開始與現實世界互動。預測市場就是一個重要的例子。 我預計預測市場將在未來幾年間，成為一個越來越重要的ethereum應用。2020年的選舉只是一個開始，我預計未來人們對預測市場會有更多的興趣，不僅僅是選舉，還有條件預測、決策和其他應用。預測市場若能照數學上那般的理想運作，會帶給我們很好的願景；但同時這也會與人類現實的侷限產生衝突。希望隨著時間過去，我們能更清楚地理解，這種新的社交科技究竟能在哪些方面提供最大的價值。 dark mode toggle christmas special 2019 dec 24 see all posts since it's christmas time now, and we're theoretically supposed to be enjoying ourselves and spending time with our families instead of waging endless holy wars on twitter, this blog post will offer some games that you can play with your friends that will help you have fun and at the same time understand some spooky mathematical concepts! 1.58 dimensional chess this is a variant of chess where the board is set up like this: the board is still a normal 8x8 board, but there are only 27 open squares. the other 37 squares should be covered up by checkers or go pieces or anything else to denote that they are inaccessible. the rules are the same as chess, with a few exceptions: white pawns move up, black pawns move left. white pawns take going left-and-up or right-and-up, black pawns take going left-and-down or left-and-up. white pawns promote upon reaching the top, black pawns promote upon reaching the left. no en passant, castling, or two-step-forward pawn jumps. chess pieces cannot move onto or through the 37 covered squares. knights cannot move onto the 37 covered squares, but don't care what they move "through". the game is called 1.58 dimensional chess because the 27 open squares are chosen according to a pattern based on the sierpinski triangle. you start off with a single open square, and then every time you double the width, you take the shape at the end of the previous step, and copy it to the top left, top right and bottom left corners, but leave the bottom right corner inaccessible. whereas in a one-dimensional structure, doubling the width increases the space by 2x, and in a two-dimensional structure, doubling the width increases the space by 4x (4 = 22), and in a three-dimensional structure, doubling the width increases the space by 8x (8 = 23), here doubling the width increases the space by 3x (3 = 21.58496), hence "1.58 dimensional" (see hausdorff dimension for details). the game is substantially simpler and more "tractable" than full-on chess, and it's an interesting exercise in showing how in lower-dimensional spaces defense becomes much easier than offense. note that the relative value of different pieces may change here, and new kinds of endings become possible (eg. you can checkmate with just a bishop). 3 dimensional tic tac toe the goal here is to get 4 in a straight line, where the line can go in any direction, along an axis or diagonal, including between planes. for example in this configuration x wins: it's considerably harder than traditional 2d tic tac toe, and hopefully much more fun! modular tic-tac-toe here, we go back down to having two dimensions, except we allow lines to wrap around: x wins note that we allow diagonal lines with any slope, as long as they pass through all four points. particularly, this means that lines with slope +/2 and +/1/2 are admissible: mathematically, the board can be interpreted as a 2-dimensional vector space over integers modulo 4, and the goal being to fill in a line that passes through four points over this space. note that there exists at least one line passing through any two points. tic tac toe over the 4-element binary field here, we have the same concept as above, except we use an even spookier mathematical structure, the 4-element field of polynomials over \(z_2\) modulo \(x^2 + x + 1\). this structure has pretty much no reasonable geometric interpretation, so i'll just give you the addition and multiplication tables: ok fine, here are all possible lines, excluding the horizontal and the vertical lines (which are also admissible) for brevity: the lack of geometric interpretation does make the game harder to play; you pretty much have to memorize the twenty winning combinations, though note that they are basically rotations and reflections of the same four basic shapes (axial line, diagonal line, diagonal line starting in the middle, that weird thing that doesn't look like a line). now play 1.77 dimensional connect four. i dare you. modular poker everyone is dealt five (you can use whatever variant poker rules you want here in terms of how these cards are dealt and whether or not players have the right to swap cards out). the cards are interpreted as: jack = 11, queen = 12, king = 0, ace = 1. a hand is stronger than another hand, if it contains a longer sequence, with any constant difference between consecutive cards (allowing wraparound), than the other hand. mathametically, this can be represented as, a hand is stronger if the player can come up with a line \(l(x) = mx+b\) such that they have cards for the numbers \(l(0)\), \(l(1)\) ... \(l(k)\) for the highest \(k\). example of a full five-card winning hand. y = 4x + 5. to break ties between equal maximum-length sequences, count the number of distinct length-three sequences they have; the hand with more distinct length-three sequences wins. this hand has four length-three sequences: k 2 4, k 4 8, 2 3 4, 3 8 k. this is rare. only consider lines of length three or higher. if a hand has three or more of the same denomination, that counts as a sequence, but if a hand has two of the same denomination, any sequences passing through that denomination only count as one sequence. this hand has no length-three sequences. if two hands are completely tied, the hand with the higher highest card (using j = 11, q = 12, k = 0, a = 1 as above) wins. enjoy! dark mode toggle 收入-邪恶曲线：思考“公共物品融资优先”的另一种方式 2022 oct 28 see all posts 译者：龙犄角 & 审阅：carol @ greenpill cn 原文：../../../2022/10/28/revenue_evil.html 特别感谢 karl floersch、hasu 和 tina zhen 对本篇内容的反馈与评论。 在任何大规模的生态中，公共物品都是无比重要的主题。但同时，"公共物品"也令人惊讶地难以定义。经济学家对"公共物品"是这样定义的：公共物品是那些非排他也非竞争性的物品。"非排他"和"非竞争"两个经济学专业术语合在一起的表述，意味着从私人财产和市场的角度来提供公共物品是十分困难的。门外汉们对于"公共物品"也有一个定义："一切对公共有益处的事物都是公共物品"。同时，民主人士对"公共物品"也提出了一个定义，为之添加了公众参与决策的含义。 然而，更重要的是，当抽象的非排他性非竞争性公共物品在与真实的世界互动时，几乎每一个具体的例子都包含无数微妙的边缘案例，需要因地制宜、具体案例具体分析。例如，一座公园是一个公共物品。但假如你对每个进入公园的人都收取 5 美金的门票钱呢？要是你通过拍卖在公园中心广场上树立价高者雕像的权力来为其募资呢？再或者这座公园是由一名半利他主义的亿万富丰出资建造、修葺、维护的，她享受这座公园作为她的个人用途、她将这座公园设计成自己喜欢的模样，但同时又允许任何路过的人踏进公园小坐一会儿、抑或和友人并肩散步谈天呢？ 这篇内容将尝试为落在"私人-公共"这条光谱上的"混合"物品进行分析提供另一个角度，也就是****收入-邪恶曲线****。我们抛出这样一个问题：不同方式的货币化项目的权衡是什么，以及通过增加外部补贴来消除货币化的压力可以做多少好事？这远不同于一个普世的框架：它假定在一个单一的"社区"中存在"混合经济"环境，有一个商业市场和一个中央资助者的补贴。但它仍旧能够在"如何在如今的加密社区、加密国家和许多其他现实世界背景下为公共产品提供资金"这方面教会我们很多道理。 传统框架：排他性和竞争性 让我们先从这里理解这个问题开始：通常的经济学家是如何看待一个项目是否属于私人物品或是公共物品的？考虑下面的例子： alice 持有 1000 以太，想要在市场上抛售。 bob 经营一家航空公司，出售航班机票。 charlie 建了一座桥，并通过收取过路费来支付这座桥的修建费用。 david 制作并发布了一档播客。 eve 制作并出了一首歌。 fred 发明了一种新的、更好的加密算法，用于进行零知识证明。 接下来，我们将这些案例放进一个有两条坐标轴的图表： 竞争性：一个人享受了美好事物的时候，她会在多大程度上降低另一个人享受美好事物的能力？ 排他性：阻止特定的人——例如那些不付钱的人——享受美好的事物，有多难？ 一张类似的图表大概长这样： 艾莉丝持有的以太是完全排他的（她对"谁可以得到她的以太有完全的掌控和选择权），而加密货币是竞争性的（如果一个人拥有一个特定的硬币，其他人就不会拥有同样的硬币）。 鲍勃的飞机票是具有排他性的，但有一点没那么具有竞争性：很可能飞机没有满座。 查理的桥比起鲍勃的飞机票更少了一点排他性，因为增加一道关卡来验证缴费情况需要额外的工作（所以查理可以排他，但对他和过路的人来说都是昂贵的），且桥的竞争性取决于道路是否拥堵。 戴卫的播客和尹芙的歌不具有竞争性：一个人听播客或者那首歌并不妨碍另一个人做同样的事情。它们倒有一点排他性，因为戴卫和尹芙可以做一个付费墙，不过人们同样可以绕过付费墙。 最后，福拉德发明的加密算法接近于完全没有排他性：算法需要开源，人们才会信任它。如果福拉德试图为算法申请专利，目标用户群（热爱开源的加密用户）很可能拒绝使用该算法，甚至为此取消他的资格。 以上都是很好也很重要的分析。排他性告诉我们你是否能够通过收费作为一种商业模式来资助这个项目，而竞争性则告诉我们排他性是否是一种悲剧性的浪费。或者说，这是否只是有关物品的一个不可避免的属性，即如果一个人得到了它，另一个人就得不到。但如果我们仔细地思考其中的一些例子，尤其是数字相关的例子，我们可以发现，这其中忽略了一个非常重要的问题：除了排他性之外，还有很多商业模式可供选择，这些商业模式也有取舍。 思考这样一个特殊的案例：戴卫的播客和尹芙的歌。在现实中，大量的播客是基本或完全免费发布的，但歌曲更多是以许可和版权限制为门槛。要知道个中原因，我们只需要明白这些播客是如何得到资助的：赞助。播客主持人通常寻找几个赞助人，并在每一集播客的开始或中间简短地为赞助人打一打广告。但赞助歌曲更难：你不能在唱一首缱绻的情歌时突兀地中途停下来并开始讲起 athletic greens* 如何如何棒呆。因为，你可拉倒吧，气氛都让你给搅合没了啊兄弟！ 我们是否能够越过仅仅考虑排他性，去更广泛地讨论货币化和不同货币化策略的危害呢？实际上，我们的确可以，而这也恰恰是收入/邪恶曲线的用处。 收入-邪恶曲线的定义 一件产品的收入-邪恶曲线是一条二维曲线，能够画出以下问题的答案： 该产品的创造者要对她们的潜在用户和更广泛的社区造成多大的伤害，才能赚取 n 美金的收入来支付产品的建设费用？ 在这里用"邪恶"这个词并不是说任何数量的邪恶都不被接受，也不是说如果你不能保证不作恶，你就根本不应该去资助项目。许多项目为了确保可持续的资金，做出了艰难的取舍，伤害了她们的客户和社区，而且往往项目存在的价值大大超过了这些伤害。但尽管如此，我们的目标是强调许多货币化计划有其悲剧性的一面，而公共物品融资可以提供价值，给现有项目一个财政缓冲，使她们能够避免这种牺牲。 下图是先前 6 个例子的收入-邪恶曲线的粗略绘制： 对于 alice 来说，以市场价格出售她的以太实际上是她能做的最有同情心的事情。如果她卖得更便宜，她几乎一定会造成链上 gas 战争、交易员 hft 战争或其他类似的价值破坏性金融冲突，因为每个人都想最快地得到她的币。而高于市场价格出售甚至不足以成为一种选择：没有人会买。 对于 bob 来说，社会最优的售价是所有机票都能卖完的最高价格。如果 bob 以低于这个价格的票价出售机票，那么票很快就会卖完，一些真正需要这些机票的人根本无法买到座位（定价过低可能会给穷人带来一些相反的好处，但也远远不是实现这一目标的最有效方式）。bob 也可以高于市场价格出售，并有可能赚取更高的利润，但代价是卖出更少的座位，而且（从上帝的角度来看）不必要地把人排除在外了。 如果 charlie 的桥和通往该桥的道路不拥堵，收取任何费用都是一种负担，并不必要地将司机排除在外。如果它们是拥堵的，低收费有助于减少拥堵，而高收费则不必要地将人们排除在外。 david 的播客可以通过增加赞助商的广告而在一定程度上实现货币化，同时也不会对听众造成很大伤害。如果货币化的压力增加，david 将不得不采用越来越多的广告形式，而真正的收入最大化将需要对播客进行付费墙屏蔽，这对潜在的听众来说是一种高成本。 eve 的处境和 david 差不多，但低伤害的选择较少（也许是卖 nft？） 。特别是在卖歌的情况下，付费墙很可能需要积极参与版权执法和起诉侵权者的法律机制，这带来了进一步的伤害。 fred 的货币化选择甚至更少。他可以为它申请专利，或者有可能采取一些怪异的举动，例如拍卖选择参数的权利，让那些喜欢特定价值的硬件制造商来竞标。但所有的选择都是高成本的。 从这里我们可以看到，在收入-邪恶曲线上实际有许多种"邪恶"。 传统的排他性经济损失：如果一个产品的价格高于边际成本，本可以进行的互利交易就不会发生。 竞争条件：拥挤、短缺和因为产品太便宜而产生的其他成本。 "污染"产品，使其对赞助商有吸引力，但对听众有一定程度的伤害（伤害或大或小）。 通过法律系统从事攻击性行动，这增加了每个人的恐惧和花钱请律师的需要，并产生各种难以预测的次生寒蝉效应。这种情况在专利方面尤为严重。 在用户、社区，甚至是从事项目本身的人高度重视的原则上做出牺牲。 在许多案例中，这种邪恶非常依赖于背景。在加密领域和更广泛的软件领域，专利权既极为有害，又具有意识形态上的攻击性，但在制造实物商品的行业中，这种情况较少：在有形商品行业中，大多数人在现实中能够创造出某种专利的衍生作品，她们的规模和组织程度足以谈判获得许可，而资本成本意味着对货币化的需求要高得多，因此保持纯粹性就更难。广告多么有害取决于广告商和受众：如果播客主持人非常了解她的听众，广告甚至可以是有帮助的！是否存在"排他"的可能性甚至取决于产权。 但通过笼统地谈论"为赚取收入而作恶"这个话题，我们获得了将这些情况相互比较的能力。 收入-邪恶曲线在如何确定融资的优先次序上告诉了我们什么？ 现在，回到我们为什么关心什么是公共物品、什么不是公共物品的关键问题：融资的优先次序。如果我们有一个有限的资本池，专门用于帮助一个社区的繁荣，我们应该把资金用于哪些方面？收入-邪恶曲线图为我们提供了回答这个问题的一个简单的起点：将资金投向那些收入-邪恶曲线斜率最大的项目。 我们应该把重点放在那些通过减少货币化的压力，每 1 美金的补贴都能最大幅度地减少不幸的项目所需的罪恶的项目上。这样我们就得到了如下的大致排名： 最优先的是 "纯粹的"公共物品，因为往往根本没有任何方法可以将其货币化，或者即使有，试图货币化的经济或道德成本也非常高。 第二个优先事项是 "自然而然"公共但可盈利的物品，通过稍加调整就可以通过商业渠道获得资金，如歌曲或对播客的赞助。 第三优先考虑的是非商品类私人物品，在这些物品中，社会福利已经通过收费得到优化，但利润率很高，或者更普遍的是存在"污染"产品以增加收入的机会。例如，通过保持伴随的软件闭源或拒绝使用标准，补贴可以用来推动这些项目在边际上做出更亲社会的选择。 需要注意的是，排他性和竞争性框架通常会得出类似的答案：首先关注非排他性和非竞争性的物品，其次关注具有排他性但非竞争性的物品，最后关注具有排他性和部分竞争性的物品——而既具有排他性又具有竞争性的物品永远不会被关注（如果你有剩余的资本，最好直接作为 ubi 发放）。收入/邪恶曲线与排他性和竞争性之间有一个粗略的近似映射：排他性越高意味着收入/邪恶曲线的斜率越低，而竞争性则告诉我们收入/邪恶曲线的底部是零还是非零。但收入/邪恶曲线是一个更普遍的工具，它允许我们谈论货币化战略的权衡，远远超出了排他。 这个框架如何用于分析决策的一个实例是 wikimedia 捐款。我个人从未向 wikimedia 捐款，因为我一直认为她们能够也应该不依靠有限的公益资金来资助自己，只需增加一些广告。而这对她们的用户体验和中立性来说只是一个很小的代价。然而，维基百科的管理员不同意；她们甚至有一个维基页面列出了她们不同意的论据。 我们可以把这种分歧理解为对收入-邪恶曲线的争议：我认为 wikimedia 的收入-邪恶曲线的斜率很低（"广告并不那么糟糕"），因此她们对我的慈善资金来说是低优先级的；其他一些人认为她们的收入-邪恶曲线的斜率很高，因此她们的慈善资金对 wikimedia 是高度优先的。 收入-邪恶曲线是一个智力工具，而「不是」一个好的直接机制 从这个想法中得出的一个重要结论是，我们不应该尝试直接使用收入-邪恶曲线作为确定单个项目优先次序的方法。由于监测方面的限制，我们在这方面的能力受到了严重制约。 如果这个框架被广泛使用，项目就会有动力去虚报她们的收入-邪恶曲线。任何收取通行费的人都有动力想出巧妙的论据，试图说明如果通行费能降低 20%，世界就会好得多，但由于她们的预算严重不足，没有补贴她们就无法降低通行费。项目将有动力在短期内更加邪恶，以吸引补贴，帮助她们变得不那么邪恶。 由于这些原因，最好的办法可能是不把该框架作为直接分配决策的方式，而是确定哪些类型的项目应优先获得资金的一般原则。例如，该框架可以成为确定如何对整个行业或整个类别的物品进行优先排序的有效方式。它可以帮助你回答这样的问题：如果一家公司正在生产一种公共物品，或者在设计一种不完全是公共物品的物品的过程中做出了有利于社会但经济成本很高的选择，她们是否应该为此获得补贴？但即使是这样，最好也是把收入-邪恶曲线当作一种心理工具，而不是试图精确地测量它们，并利用它们来作出单独的决定。 结论 排他性和竞争性对一个物品来说是重要的维度，对其货币化的能力，以及对回答"通过一些公共资金获得资助，可以避免多少伤害"的问题，具有真正重要的影响。但是，特别是一旦更复杂的项目进场，这两个维度很快就开始变得不足以决定资金的优先次序。大多数的物品都不是纯粹的公共物品：它们是处于中间的一些混合体，在许多方面，它们或多或少都是公共的，且不容易映射到"排他"。 观察一个项目的收入-邪恶曲线给了我们另一种衡量真正重要的统计数据的方法：减轻一个项目 1 美金的货币化压力，可以避免多少伤害？有时，缓解货币化压力的收益是决定性的：就是没有办法通过商业渠道来资助某些种类的东西，直到你能找到一个从它们中受益的单一用户，足以单方面资助它们。其他时候，商业资金的选择是存在的，但有着有害的副作用。有时这些影响较小，有时较大。有时，个人项目的一小部分在亲社会的选择和增加货币化之间有明显的权衡。还有一些时候，项目只是自己筹资，没有必要对其进行补贴——或者，至少不确定性和隐藏的信息使得创建一个利大于弊的补贴时间表太难。按照从最大收益到最小收益的顺序排列资金的优先次序总是更好的；而你能走多远取决于你有多少资金。 * 我没有接受athletic greens的赞助费用。但播客主持人 lex fridman 接受了。以及，我也没有接受 lex fridman 的赞助费用。但也许有人接受了。反正管他呢，只要我们能继续为播客提供资金，让人们能免费收听，又不至于太烦，那一切就都很好了，对吧？ jeff’s ethereum ðξv update ii | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search jeff’s ethereum ðξv update ii posted by jeffrey wilcke on january 6, 2015 research & development first of all, happy new year! what a year it has been. with a little luck we'll surpass last year with an even more awesome year. it's been too long since i've given an update on my side of things and that of the go team and mostly due to a lack of time. i've been so incredibly busy and so many things have happened these past 2 months i've hardly had time to sit down and assess it all. as you may be well aware the audit is looming around the corner and my little baby (go-ethereum!) will undergo it's full inspection very, very soon. the audit teams will tear it apart and see if the repo contains anything incorrectly implemented as well as search for any major security flaws in the design and implementation. we've been pretty solid on tests, testing implementation details as well as consensus tests (thanks to christoph) and will continue to add more tests over time. we'll see how they hold up during the audit (though i'm confident we'll be fine, it's still a little bit scary (-:) development poc-7 has been released now for a about a week and has been quite stable (and growing in size!). we're already hard at work to finalising poc-8 which includes numerous small changes: adjusted block time back to 12s (was 4s) op code prevhash has become blockhash( n ) and therefore prevhash = blockhash(number 1) we've added an additional pre-compiled contract at address 0x04 which returns the given input (acts like copy / memcpy) ongoing p2p felix has been hard at work on our new p2p package which has now entered in to v0.1 (poc-7) and will soon already undergo it's first upgrade for poc-8. felix has done an amazing job on the design of the package and it's a real pleasure to work with. auto-generated documentation can be found at godoc. whisper a month or so back i finished the first draft of whisper for the go implementation and it's now passing whisper messages nicely around the network and uses the p2p package mentioned earlier. the go api is relatively easy and requires almost zero setup. backend the backend stack of ethereum has also received its first major (well deserved) overhaul. viktor's been incredibly hard at work to reimplement the download manager and the ethereum sub protocol. swarm since the first day dani joined the team he's passionately been working on the peer selection algorithm and distributed preimage archive. the dpa will be used for our swarm tech. the spec is about 95% complete and roughly about 50% has been implemented. progress is going strong! both go-ethereum/p2p and go-ethereum/whisper have been developed in such a way that neither require ethereum to operate. if you're developing in go and your application requires a p2p network or (dark) messaging try out the packages. an example sub protocol can be found here and an example on how to use whisper can be found here. ams hub now that the hub is finally set up you're free to drop by and grab a coffee with us. you can find us in the rather posh neighbourhood of amsterdam zuid near museumplein (alexander boerstraat 21). in my next post i hope i'll have a release candidate for poc-8 and perhaps even a draft implementation of swarm. but until then, happy whispering and mining! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle travel time ~= 750 * distance ^ 0.6 2023 apr 14 see all posts as another exercise in using chatgpt 3.5 to do weird things and seeing what happens, i decided to explore an interesting question: how does the time it takes to travel from point a to point b scale with distance, in the real world? that is to say, if you sample randomly from positions where people are actually at (so, for example, 56% of points you choose would be in cities), and you use public transportation, how does travel time scale with distance? obviously, travel time would grow slower than linearly: the further you have to go, the more opportunity you have to resort to forms of transportation that are faster, but have some fixed overhead. outside of a very few lucky cases, there is no practical way to take a bus to go faster if your destination is 170 meters away, but if your destination is 170 kilometers away, you suddenly get more options. and if it's 1700 kilometers away, you get airplanes. so i asked chatgpt for the ingredients i would need: i went with the geolife dataset. i did notice that while it claims to be about users around the world, primarily it seems to focus on people in seattle and beijing, though they do occasionally visit other cities. that said, i'm not a perfectionist and i was fine with it. i asked chatgpt to write me a script to interpret the dataset and extract a randomly selected coordinate from each file: amazingly, it almost succeeded on the first try. it did make the mistake of assuming every item in the list would be a number (values = [float(x) for x in line.strip().split(',')]), though perhaps to some extent that was my fault: when i said "the first two values" it probably interpreted that as implying that the entire line was made up of "values" (ie. numbers). i fixed the bug manually. now, i have a way to get some randomly selected points where people are at, and i have an api to get the public transit travel time between the points. i asked it for more coding help: asking how to get an api key for the google maps directions api (it gave an answer that seems to be outdated, but that succeeded at immediately pointing me to the right place) writing a function to compute the straight-line distance between two gps coordinates (it gave the correct answer on the first try) given a list of (distance, time) pairs, drawing a scatter plot, with time and distance as axes, both axes logarithmically scaled (it gave the correct answer on the first try) doing a linear regression on the logarithms of distance and time to try to fit the data to a power law (it bugged on the first try, succeeded on the second) this gave me some really nice data (this is filtered for distances under 500km, as above 500km the best path almost certainly includes flying, and the google maps directions don't take into account flights): the power law fit that the linear regression gave is: travel_time = 965.8020738916074 * distance^0.6138556361612214 (time in seconds, distance in km). now, i needed travel time data for longer distances, where the optimal route would include flights. here, apis could not help me: i asked chatgpt if there were apis that could do such a thing, and it did not give a satisfactory answer. i resorted to doing it manually: i used the same script, but modified it slightly to only output pairs of points which were more than 500km apart from each other. i took the first 8 results within the united states, and the first 8 with at least one end outside the united states, skipping over results that represented a city pair that had already been covered. for each result i manually obtained: to_airport: the public transit travel time from the starting point to the nearest airport, using google maps outside china and baidu maps inside china. from_airport: the public transit travel time to the end point from the nearest airport flight_time: the flight time from the starting point to the end point. i used google flights) and always took the top result, except in cases where the top result was completely crazy (more than 2x the length of the shortest), in which case i took the shortest. i computed the travel time as (to_airport) * 1.5 + (90 if international else 60) + flight_time + from_airport. the first part is a fairly aggressive formula (i personally am much more conservative than this) for when to leave for the airport: aim to arrive 60 min before if domestic and 90 min before if international, and multiply expected travel time by 1.5x in case there are any mishaps or delays. this was boring and i was not interested in wasting my time to do more than 16 of these; i presume if i was a serious researcher i would already have an account set up on taskrabbit or some similar service that would make it easier to hire other people to do this for me and get much more data. in any case, 16 is enough; i put my resulting data here. finally, just for fun, i added some data for how long it would take to travel to various locations in space: the moon (i added 12 hours to the time to take into account an average person's travel time to the launch site), mars, pluto and alpha centauri. you can find my complete code here. here's the resulting chart: travel_time = 733.002223593754 * distance^0.591980777827876 waaaaat?!?!! from this chart it seems like there is a surprisingly precise relationship governing travel time from point a to point b that somehow holds across such radically different transit media as walking, subways and buses, airplanes and (!!) interplanetary and hypothetical interstellar spacecraft. i swear that i am not cherrypicking; i did not throw out any data that was inconvenient, everything (including the space stuff) that i checked i put on the chart. chatgpt 3.5 worked impressively well this time; it certainly stumbled and fell much less than my previous misadventure, where i tried to get it to help me convert ipfs bafyhashes into hex. in general, chatgpt seems uniquely good at teaching me about libraries and apis i've never heard of before but that other people use all the time; this reduces the barrier to entry between amateurs and professionals which seems like a very positive thing. so there we go, there seems to be some kind of really weird fractal law of travel time. of course, different transit technologies could change this relationship: if you replace public transit with cars and commercial flights with private jets, travel time becomes somewhat more linear. and once we upload our minds onto computer hardware, we'll be able to travel to alpha centauri on much crazier vehicles like ultralight craft propelled by earth-based lightsails) that could let us go anywhere at a significant fraction of the speed of light. but for now, it does seem like there is a strangely consistent relationship that puts time much closer to the square root of distance. the ethereum development process | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the ethereum development process posted by gavin wood on march 5, 2015 research & development so i'm not sure if this kind of development methodology has ever been applied to such an extreme before so i figured i'd document it. in a nutshell, it's sort of like test-driven triplet-programming development. while speed-developing our alpha codebase, four of us sat around a table in the office in berlin. three people (vitalik, jeff and me) each coders of their own clean-room implementation of the ethereum protocol. the fourth was christoph, our master of testing. our target was to have three fully compatible implementations as well as an unambiguous specification by the end of three days of substantial development. over distance, this process normally takes a few weeks. this time we needed to expedite it; our process was quite simple. first we discuss the various consensus-breaking changes and formally describe them as best we can. then, individually we each crack on coding up the changes simultaneously, popping our heads up about possible clarifications to the specifications as needed. meanwhile, christoph devises and codes tests, populating the results either manually or with the farthest-ahead of the implementations (c++, generally :-p). after a milestone's worth of changes are coded up and the tests written, each clean-room implementation is tested against the common test data that christoph compiled. where issues are found, we debug in a group. so far, this has proved to be an effective way of producing well-tested code quickly, and perhaps more importantly, in delivering clear unambiguous formal specifications. are there any more examples of such techniques taken to the extreme? previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle the bulldozer vs vetocracy political axis 2021 dec 19 see all posts typically, attempts to collapse down political preferences into a few dimensions focus on two primary dimensions: "authoritarian vs libertarian" and "left vs right". you've probably seen political compasses like this: there have been many variations on this, and even an entire subreddit dedicated to memes based on these charts. i even made a spin on the concept myself, with this "meta-political compass" where at each point on the compass there is a smaller compass depicting what the people at that point on the compass see the axes of the compass as being. of course, "authoritarian vs libertarian" and "left vs right" are both incredibly un-nuanced gross oversimplifications. but us puny-brained human beings do not have the capacity to run anything close to accurate simulations of humanity inside our heads, and so sometimes incredibly un-nuanced gross oversimplifications are something we need to understand the world. but what if there are other incredibly un-nuanced gross oversimplifications worth exploring? enter the bulldozer vs vetocracy divide let us consider a political axis defined by these two opposing poles: bulldozer: single actors can do important and meaningful, but potentially risky and disruptive, things without asking for permission vetocracy: doing anything potentially disruptive and controversial requires getting a sign-off from a large number of different and diverse actors, any of whom could stop it note that this is not the same as either authoritarian vs libertarian or left vs right. you can have vetocratic authoritarianism, the bulldozer left, or any other combination. here are a few examples: the key difference between authoritarian bulldozer and authoritarian vetocracy is this: is the government more likely to fail by doing bad things or by preventing good things from happening? similarly for libertarian bulldozer vs vetocracy: are private actors more likely to fail by doing bad things, or by standing in the way of needed good things? sometimes, i hear people complaining that eg. the united states (but other countries too) is falling behind because too many people use freedom as an excuse to prevent needed reforms from happening. but is the problem really freedom? isn't, say, restrictive housing policy preventing gdp from rising by 36% an example of the problem precisely being people not having enough freedom to build structures on their own land? shifting the argument over to saying that there is too much vetocracy, on the other hand, makes the argument look much less confusing: individuals excessively blocking governments and governments excessively blocking individuals are not opposites, but rather two sides of the same coin. and indeed, recently there has been a bunch of political writing pointing the finger straight at vetocracy as a source of many huge problems: https://astralcodexten.substack.com/p/ezra-klein-on-vetocracy https://www.vox.com/2020/4/22/21228469/marc-andreessen-build-government-coronavirus https://www.vox.com/2016/10/26/13352946/francis-fukuyama-ezra-klein https://www.politico.com/news/magazine/2019/11/29/penn-station-robert-caro-073564 and on the other side of the coin, people are often confused when politicians who normally do not respect human rights suddenly appear very pro-freedom in their love of bitcoin. are they libertarian, or are they authoritarian? in this framework, the answer is simple: they're bulldozers, with all the benefits and risks that that side of the spectrum brings. what is vetocracy good for? though the change that cryptocurrency proponents seek to bring to the world is often bulldozery, cryptocurrency governance internally is often quite vetocratic. bitcoin governance famously makes it very difficult to make changes, and some core "constitutional norms" (eg. the 21 million coin limit) are considered so inviolate that many bitcoin users consider a chain that violates that rule to be by-definition not bitcoin, regardless of how much support it has. ethereum protocol research is sometimes bulldozery in operation, but the ethereum eip process that governs the final stage of turning a research proposal into something that actually makes it into the blockchain includes a fair share of vetocracy, though still less than bitcoin. governance over irregular state changes, hard forks that interfere with the operation of specific applications on-chain, is even more vetocratic: after the dao fork, not a single proposal to intentionally "fix" some application by altering its code or moving its balance has been successful. the case for vetocracy in these contexts is clear: it gives people a feeling of safety that the platform they build or invest on is not going to suddenly change the rules on them one day and destroy everything they've put years of their time or money into. cryptocurrency proponents often cite citadel interfering in gamestop trading as an example of the opaque, centralized (and bulldozery) manipulation that they are fighting against. web2 developers often complain about centralized platforms suddenly changing their apis in ways that destroy startups built around their platforms. and, of course.... vitalik buterin, bulldozer victim ok fine, the story that wow removing siphon life was the direct inspiration to ethereum is exaggerated, but the infamous patch that ruined my beloved warlock and my response to it were very real! and similarly, the case for vetocracy in politics is clear: it's a response to the often ruinous excesses of the bulldozers, both relatively minor and unthinkably severe, of the early 20th century. so what's the synthesis? the primary purpose of this point is to outline an axis, not to argue for a particular position. and if the vetocracy vs bulldozer axis is anything like the libertarian vs authoritarian axis, it's inevitably going to have internal subtleties and contradictions: much like a free society will see people voluntarily joining internally autocratic corporations (yes, even lots of people who are totally not economically desperate make such choices), many movements will be vetocratic internally but bulldozery in their relationship with the outside world. but here are a few possible things that one could believe about bulldozers and vetocracy: the physical world has too much vetocracy, but the digital world has too many bulldozers, and there are no digital places that are truly effective refuges from the bulldozers (hence: why we need blockchains?) processes that create durable change need to be bulldozery toward the status quo but protecting that change requires a vetocracy. there's some optimal rate at which such processes should happen; too much and there's chaos, not enough and there's stagnation. a few key institutions should be protected by strong vetocracy, and these institutions exist both to enable bulldozers needed to enact positive change and to give people things they can depend on that are not going to be brought down by bulldozers. in particular, blockchain base layers should be vetocratic, but application-layer governance should leave more space for bulldozers better economic mechanisms (quadratic voting? harberger taxes?) can get us many of the benefits of both vetocracy and bulldozers without many of the costs. vetocracy vs bulldozer is a particularly useful axis to use when thinking about non-governmental forms of human organization, whether for-profit companies, non-profit organizations, blockchains, or something else entirely. the relatively easier ability to exit from such systems (compared to governments) confounds discussion of how libertarian vs authoritarian they are, and so far blockchains and even centralized tech platforms have not really found many ways to differentiate themselves on the left vs right axis (though i would love to see more attempts at left-leaning crypto projects!). the vetocracy vs bulldozer axis, on the other hand, continues to map to non-governmental structures quite well potentially making it very relevant in discussing these new kinds of non-governmental structures that are becoming increasingly important. dark mode toggle a quick garbled circuits primer 2020 mar 21 see all posts special thanks to dankrad feist for review garbled circuits are a quite old, and surprisingly simple, cryptographic primitive; they are quite possibly the simplest form of general-purpose "multi-party computation" (mpc) to wrap your head around. here is the usual setup for the scheme: suppose that there are two parties, alice and bob, who want to compute some function f(alice_inputs, bob_inputs), which takes inputs from both parties. alice and bob want to both learn the result of computing f, but alice does not want bob to learn her inputs, and bob does not want alice to learn his inputs. ideally, they would both learn nothing except for just the output of f. alice performs a special procedure ("garbling") to encrypt a circuit (meaning, a set of and, or... gates) which evaluates the function f. she passes along inputs, also encrypted in a way that's compatible with the encrypted circuit, to bob. bob uses a technique called "1-of-2 oblivious transfer" to learn the encrypted form of his own inputs, without letting alice know which inputs he obtained. bob runs the encrypted circuit on the encrypted data and gets the answer, and passes it along to alice. extra cryptographic wrappings can be used to protect the scheme against alice and bob sending wrong info and giving each other an incorrect answer; we won't go into those here for simplicity, though it suffices to say "wrap a zk-snark around everything" is one (quite heavy duty and suboptimal!) solution that works fine. so how does the basic scheme work? let's start with a circuit: this is one of the simplest examples of a not-completely-trivial circuit that actually does something: it's a two-bit adder. it takes as input two numbers in binary, each with two bits, and outputs the three-bit binary number that is the sum. now, let's encrypt the circuit. first, for every input, we randomly generate two "labels" (think: 256-bit numbers): one to represent that input being 0 and the other to represent that input being 1. then we also do the same for every intermediate wire, not including the output wires. note that this data is not part of the "garbling" that alice sends to bob; so far this is just setup. now, for every gate in the circuit, we do the following. for every combination of inputs, we include in the "garbling" that alice provides to bob the label of the output (or if the label of the output is a "final" output, the output directly) encrypted with a key generated by hashing the input labels that lead to that output together. for simplicity, our encryption algorithm can just be enc(out, in1, in2) = out + hash(k, in1, in2) where k is the index of the gate (is it the first gate in the circuit, the second, the third?). if you know the labels of both inputs, and you have the garbling, then you can learn the label of the corresponding output, because you can just compute the corresponding hash and subtract it out. here's the garbling of the first xor gate: inputs output encoding of output 00 0 0 + hash(1, 6816, 6529) 01 1 1 + hash(1, 6816, 4872) 10 1 1 + hash(1, 8677, 6529) 11 0 0 + hash(1, 8677, 4872) notice that we are including the (encrypted forms of) 0 and 1 directly, because this xor gate's outputs are directly final outputs of the program. now, let's look at the leftmost and gate: inputs output encoding of output 00 0 5990 + hash(2, 6816, 6529) 01 0 5990 + hash(2, 6816, 4872) 10 0 5990 + hash(2, 8677, 6529) 11 1 1921 + hash(2, 8677, 4872) here, the gate's outputs are just used as inputs to other gates, so we use labels instead of bits to hide these intermediate bits from the evaluator. the "garbling" that alice would provide to bob is just everything in the third column for each gate, with the rows of each gate re-ordered (to avoid revealing whether a given row corresponds to a 0 or a 1 in any wire). to help bob learn which value to decrypt for each gate, we'll use a particular order: for each gate, the first row becomes the row where both input labels are even, in the second row the second label is odd, in the third row the first label is odd, and in the fourth row both labels are odd (we deliberately chose labels earlier so that each gate would have an even label for one output and an odd label for the other). we garble every other gate in the circuit in the same way. all in all, alice sends to bob four ~256 bit numbers for each gate in the circuit. it turns out that four is far from optimal; see here for some optimizations on how to reduce this to three or even two numbers for an and gate and zero (!!) for an xor gate. note that these optimizations do rely on some changes, eg. using xor instead of addition and subtraction, though this should be done anyway for security. when bob receives the circuit, he asks alice for the labels corresponding to her input, and he uses a protocol called "1-of-2 oblivious transfer" to ask alice for the labels corresponding to his own input without revealing to alice what his input is. he then goes through the gates in the circuit one by one, uncovering the output wires of each intermediate gate. suppose alice's input is the two left wires and she gives (0, 1), and bob's input is the two right wires and he gives (1, 1). here's the circuit with labels again: at the start, bob knows the labels 6816, 3621, 4872, 5851 bob evaluates the first gate. he knows 6816 and 4872, so he can extract the output value corresponding to (1, 6816, 4872) (see the table above) and extracts the first output bit, 1 bob evaluates the second gate. he knows 6816 and 4872, so he can extract the output value corresponding to (2, 6816, 4872) (see the table above) and extracts the label 5990 bob evaluates the third gate (xor). he knows 3621 and 5851, and learns 7504 bob evaluates the fourth gate (or). he knows 3621 and 5851, and learns 6638 bob evaluates the fifth gate (and). he knows 3621 and 5851, and learns 7684 bob evaluates the sixth gate (xor). he knows 5990 and 7504, and learns the second output bit, 0 bob evaluates the seventh gate (and). he knows 5990 and 6638, and learns 8674 bob evaluates the eighth gate (or). he knows 8674 and 7684, and learns the third output bit, 1 and so bob learns the output: 101. and in binary 10 + 11 actually equals 101 (the input and output bits are both given in smallest-to-greatest order in the circuit, which is why alice's input 10 is represented as (0, 1) in the circuit), so it worked! note that addition is a fairly pointless use of garbled circuits, because bob knowing 101 can just subtract out his own input and get 101 11 = 10 (alice's input), breaking privacy. however, in general garbled circuits can be used for computations that are not reversible, and so don't break privacy in this way (eg. one might imagine a computation where alice's input and bob's input are their answers to a personality quiz, and the output is a single bit that determines whether or not the algorithm thinks they are compatible; that one bit of information won't let alice or bob know anything about each other's individual quiz answers). 1 of 2 oblivious transfer now let us talk more about 1-of-2 oblivious transfer, this technique that bob used to obtain the labels from alice corresponding to his own input. the problem is this. focusing on bob's first input bit (the algorithm for the second input bit is the same), alice has a label corresponding to 0 (6529), and a label corresponding to 1 (4872). bob has his desired input bit: 1. bob wants to learn the correct label (4872) without letting alice know that his input bit is 1. the trivial solution (alice just sends bob both 6529 and 4872) doesn't work because alice only wants to give up one of the two input labels; if bob receives both input labels this could leak data that alice doesn't want to give up. here is a fairly simple protocol using elliptic curves: alice generates a random elliptic curve point, h. bob generates two points, p1 and p2, with the requirement that p1 + p2 sums to h. bob chooses either p1 or p2 to be g * k (ie. a point that he knows the corresponding private key for). note that the requirement that p1 + p2 = h ensures that bob has no way to generate p1 and p2 such that he knows the corresponding private key for. this is because if p1 = g * k1 and p2 = g * k2 where bob knows both k1 and k2, then h = g * (k1 + k2), so that would imply bob can extract the discrete logarithm (or "corresponding private key") for h, which would imply all of elliptic curve cryptography is broken. alice confirms p1 + p2 = h, and encrypts v1 under p1 and v2 under p2 using some standard public key encryption scheme (eg. el-gamal). bob is only able to decrypt one of the two values, because he knows the private key corresponding to at most one of (p1, p2), but alice does not know which one. this solves the problem; bob learns one of the two wire labels (either 6529 or 4872), depending on what his input bit is, and alice does not know which label bob learned. applications garbled circuits are potentially useful for many more things than just 2-of-2 computation. for example, you can use them to make multi-party computations of arbitrary complexity with an arbitrary number of participants providing inputs, that can run in a constant number of rounds of interaction. generating a garbled circuit is completely parallelizable; you don't need to finish garbling one gate before you can start garbling gates that depend on it. hence, you can simply have a large multi-party computation with many participants compute a garbling of all gates of a circuit and publish the labels corresponding to their inputs. the labels themselves are random and so reveal nothing about the inputs, but anyone can then execute the published garbled circuit and learn the output "in the clear". see here for a recent example of an mpc protocol that uses garbling as an ingredient. multi-party computation is not the only context where this technique of splitting up a computation into a parallelizable part that operates on secret data followed by a sequential part that can be run in the clear is useful, and garbled circuits are not the only technique for accomplishing this. in general, the literature on randomized encodings includes many more sophisticated techniques. this branch of math is also useful in technologies such as functional encryption and obfuscation. implementing vitalik’s vision | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search implementing vitalik’s vision posted by taylor gerring on april 2, 2015 research & development although the core concept was proven last summer when lead developers vitalik buterin, gavin wood, and jeffrey wilcke initiated the poc (proof-of-concept) series in python, c++, and go, a lot of engineering effort has gone to solidifying these implementations. we're all anxious to release a stable command line client and issues flagged in the external audit are being resolved when uncovered. as soon as this process is complete, a general release should be around the corner. in the meantime, the rest of the ecosystem is continuing to evolve to more refined versions. one of the challenges with a project of this scope and complexity is simply agreeing on any particular thing. translating human abstractions into code is a tough task, as even the words we choose influences different ways of expressing ideas with code. as we build more around these core ideas and truly experience how reality meets theory, we’re able to refine the underlying specifications to remove ambiguity. this all pays off as the team shares a clean and well-understood vision that should translate effectively to other developers of varying disciplines. number of git commits over time complementing this, a variety of test suites add an important level of confidence that the various implementations adhere to a specification. on the consensus front, we have used a series of vm and state tests to simulate known results and have added randomization and fuzzing to increase this testing space. a series of system and networking tests are also being run, ensuring clients communicate in an expected manner or respond to certain networking scenarios predictably. not only are tests designed for success, but all sorts of error conditions are fed into the test suites to ensure that known errors (such as running out of gas or being abruptly disconnected from a peer) are reported correctly. what this means is that when changes are made to the behaviour of the clients, the decision is driven by data and metrics, rather than theory and gut feelings. graphing node connectivity allows us to visually inspect the health of network beyond testing of the core components, we have begun implementing tests on the exterior layers to ensure that the clients respond to a multitude of calls to the underlying system in the same way with easily-consumable json. these specifications are linked in the wiki and drive the implementation and communication across diverse teams. this helps to ensure that what is in our heads is reliably translated into code, and that all of this is tested and audited. all of these improvements are important to the health of the overlaying ecosystem and help shine a spotlight on otherwise very nuanced problems. in addition to refining the core virtual machine and serialization layers, security audits were initiated with a third-party firm experienced in decentralization and cryptographic technologies. a rough snapshot of development efforts. so how does all of this underlying technobabble expose itself to the general user? for the frontier rollout, we are aiming to release stable binaries on major operating systems (mac, windows, linux). the main interfaces will consist primarily of the command-line client, interactive javascript console, and json-rpc. in go-ethereum, we’ve recently renamed this client to “geth”, a nod to both mass effect and old english. using the command-line client will be a familiar process to many. for example, you can create a new account with geth account new, at which point the program will prompt you for a password to secure the account. starting geth as a miner can be done by calling it with a parameter, like so: geth -mine. this is a great way to start interacting with the system, but has its own set of limitations. for example, once you’ve started mining, how does one pause it? using geth’s interactive javascript console for a more useful way to control the program, geth has a javascript console backed by a javascript api. start it with a simple command: geth console. with geth, this starts the interactive console, giving users the familiarity of javascript scripting to control the system from the terminal. for example, to find out a bit more about the local node, run admin.nodeinfo(). this control is partly powered by ethereum.js, a javascript library that helps provide a friendly interface to not only the console, but also dapps. it relies on a well-documented json-rpc interface for automated processing or integration into existing infrastructure. work on many of these components is ongoing as rigorous effort is being put in to ensuring that the implementations speak the same language and are as generally compatible as possible. a quick view of build status of course, all of this is built publicly on github and includes various continuous integration aspects such as automated builds and reporting on major platforms. this is not a program defining a spec—this is a spec implemented and cross tested between machines and humans spanning the globe. concurrently, work is progressing on several other components to the ecosystem, including gui interfaces for mist (the dapp browser) and mix (the ide). many of the tools we use and build upon are new and sometimes require patching upstream to ensure everything works as expected. this is a great benefit for all open source development, and one of the many positive byproducts of all the hours spent actually engineering software. although the wait may seem long, progress and ongoing confidence in the whole system is increasing steadily. beyond the technical ins and outs of development, our communications team has continued interacting with the community to produce public education material like tutorials and videos. furthermore, our admin staff helps process payments and comply with various local laws including tedious things like employment taxes. who needs sleep? being part of the ethereum project has been an exceptional experience filled with excitement. what started as only an idea little more than a year ago has spawned an exceptional engineering effort led by professionals implementing all the necessary infrastructure required to build a revolutionary decentralized application platform. when netflix introduced chaos monkey, they asserted, “we have found that the best defense against major unexpected failures is to fail often. by frequently causing failures, we force our services to be built in a way that is more resilient”. in the process of developing ethereum, we’ve experienced first-hand the benefits that come from continually straining a system. and while being tasked to craft the very platform that you dream about is a huge honour, we build it because we want it to exist. thanks for your patience as we think, sleep, eat, and dogfood the platform. we're as anxious as anyone to see what kind of daapps will be launching after mainnet goes live! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle an incomplete guide to rollups 2021 jan 05 see all posts rollups are all the rage in the ethereum community, and are poised to be the key scalability solution for ethereum for the foreseeable future. but what exactly is this technology, what can you expect from it and how will you be able to use it? this post will attempt to answer some of those key questions. background: what is layer-1 and layer-2 scaling? there are two ways to scale a blockchain ecosystem. first, you can make the blockchain itself have a higher transaction capacity. the main challenge with this technique is that blockchains with "bigger blocks" are inherently more difficult to verify and likely to become more centralized. to avoid such risks, developers can either increase the efficiency of client software or, more sustainably, use techniques such as sharding to allow the work of building and verifying the chain to be split up across many nodes; the effort known as "eth2" is currently building this upgrade to ethereum. second, you can change the way that you use the blockchain. instead of putting all activity on the blockchain directly, users perform the bulk of their activity off-chain in a "layer 2" protocol. there is a smart contract on-chain, which only has two tasks: processing deposits and withdrawals, and verifying proofs that everything happening off-chain is following the rules. there are multiple ways to do these proofs, but they all share the property that verifying the proofs on-chain is much cheaper than doing the original computation off-chain. state channels vs plasma vs rollups the three major types of layer-2 scaling are state channels, plasma and rollups. they are three different paradigms, with different strengths and weaknesses, and at this point we are fairly confident that all layer-2 scaling falls into roughly these three categories (though naming controversies exist at the edges, eg. see "validium"). how do channels work? see also: https://www.jeffcoleman.ca/state-channels and statechannels.org imagine that alice is offering an internet connection to bob, in exchange for bob paying her $0.001 per megabyte. instead of making a transaction for each payment, alice and bob use the following layer-2 scheme. first, bob puts $1 (or some eth or stablecoin equivalent) into a smart contract. to make his first payment to alice, bob signs a "ticket" (an off-chain message), that simply says "$0.001", and sends it to alice. to make his second payment, bob would sign another ticket that says "$0.002", and send it to alice. and so on and so forth for as many payments as needed. when alice and bob are done transacting, alice can publish the highest-value ticket to chain, wrapped in another signature from herself. the smart contract verifies alice and bob's signatures, pays alice the amount on bob's ticket and returns the rest to bob. if alice is unwilling to close the channel (due to malice or technical failure), bob can initiate a withdrawal period (eg. 7 days); if alice does not provide a ticket within that time, then bob gets all his money back. this technique is powerful: it can be adjusted to handle bidirectional payments, smart contract relationships (eg. alice and bob making a financial contract inside the channel), and composition (if alice and bob have an open channel and so do bob and charlie, alice can trustlessly interact with charlie). but there are limits to what channels can do. channels cannot be used to send funds off-chain to people who are not yet participants. channels cannot be used to represent objects that do not have a clear logical owner (eg. uniswap). and channels, especially if used to do things more complex than simple recurring payments, require a large amount of capital to be locked up. how does plasma work? see also: the original plasma paper, and plasma cash. to deposit an asset, a user sends it to the smart contract managing the plasma chain. the plasma chain assigns that asset a new unique id (eg. 537). each plasma chain has an operator (this could be a centralized actor, or a multisig, or something more complex like pos or dpos). every interval (this could be 15 seconds, or an hour, or anything in between), the operator generates a "batch" consisting of all of the plasma transactions they have received off-chain. they generate a merkle tree, where at each index x in the tree, there is a transaction transferring asset id x if such a transaction exists, and otherwise that leaf is zero. they publish the merkle root of this tree to chain. they also send the merkle branch of each index x to the current owner of that asset. to withdraw an asset, a user publishes the merkle branch of the most recent transaction sending the asset to them. the contract starts a challenge period, during which anyone can try to use other merkle branches to invalidate the exit by proving that either (i) the sender did not own the asset at the time they sent it, or (ii) they sent the asset to someone else at some later point in time. if no one proves that the exit is fraudulent for (eg.) 7 days, the user can withdraw the asset. plasma provides stronger properties than channels: you can send assets to participants who were never part of the system, and the capital requirements are much lower. but it comes at a cost: channels require no data whatsoever to go on chain during "normal operation", but plasma requires each chain to publish one hash at regular intervals. additionally, plasma transfers are not instant: you have to wait for the interval to end and for the block to be published. additionally, plasma and channels share a key weakness in common: the game theory behind why they are secure relies on the idea that each object controlled by both systems has some logical "owner". if that owner does not care about their asset, then an "invalid" outcome involving that asset may result. this is okay for many applications, but it is a deal breaker for many others (eg. uniswap). even systems where the state of an object can be changed without the owner's consent (eg. account-based systems, where you can increase someone's balance without their consent) do not work well with plasma. this all means that a large amount of "application-specific reasoning" is required in any realistic plasma or channels deployment, and it is not possible to make a plasma or channel system that just simulates the full ethereum environment (or "the evm"). to get around this problem, we get to... rollups. rollups see also: ethhub on optimistic rollups and zk rollups. plasma and channels are "full" layer 2 schemes, in that they try to move both data and computation off-chain. however, fundamental game theory issues around data availability means that it is impossible to safely do this for all applications. plasma and channels get around this by relying on an explicit notion of owners, but this prevents them from being fully general. rollups, on the other hand, are a "hybrid" layer 2 scheme. rollups move computation (and state storage) off-chain, but keep some data per transaction on-chain. to improve efficiency, they use a whole host of fancy compression tricks to replace data with computation wherever possible. the result is a system where scalability is still limited by the data bandwidth of the underlying blockchain, but at a very favorable ratio: whereas an ethereum base-layer erc20 token transfer costs ~45000 gas, an erc20 token transfer in a rollup takes up 16 bytes of on-chain space and costs under 300 gas. the fact that data is on-chain is key (note: putting data "on ipfs" does not work, because ipfs does not provide consensus on whether or not any given piece of data is available; the data must go on a blockchain). putting data on-chain and having consensus on that fact allows anyone to locally process all the operations in the rollup if they wish to, allowing them to detect fraud, initiate withdrawals, or personally start producing transaction batches. the lack of data availability issues means that a malicious or offline operator can do even less harm (eg. they cannot cause a 1 week delay), opening up a much larger design space for who has the right to publish batches and making rollups vastly easier to reason about. and most importantly, the lack of data availability issues means that there is no longer any need to map assets to owners, leading to the key reason why the ethereum community is so much more excited about rollups than previous forms of layer 2 scaling: rollups are fully general-purpose, and one can even run an evm inside a rollup, allowing existing ethereum applications to migrate to rollups with almost no need to write any new code. ok, so how exactly does a rollup work? there is a smart contract on-chain which maintains a state root: the merkle root of the state of the rollup (meaning, the account balances, contract code, etc, that are "inside" the rollup). anyone can publish a batch, a collection of transactions in a highly compressed form together with the previous state root and the new state root (the merkle root after processing the transactions). the contract checks that the previous state root in the batch matches its current state root; if it does, it switches the state root to the new state root. to support depositing and withdrawing, we add the ability to have transactions whose input or output is "outside" the rollup state. if a batch has inputs from the outside, the transaction submitting the batch needs to also transfer these assets to the rollup contract. if a batch has outputs to the outside, then upon processing the batch the smart contract initiates those withdrawals. and that's it! except for one major detail: how to do know that the post-state roots in the batches are correct? if someone can submit a batch with any post-state root with no consequences, they could just transfer all the coins inside the rollup to themselves. this question is key because there are two very different families of solutions to the problem, and these two families of solutions lead to the two flavors of rollups. optimistic rollups vs zk rollups the two types of rollups are: optimistic rollups, which use fraud proofs: the rollup contract keeps track of its entire history of state roots and the hash of each batch. if anyone discovers that one batch had an incorrect post-state root, they can publish a proof to chain, proving that the batch was computed incorrectly. the contract verifies the proof, and reverts that batch and all batches after it. zk rollups, which use validity proofs: every batch includes a cryptographic proof called a zk-snark (eg. using the plonk protocol), which proves that the post-state root is the correct result of executing the batch. no matter how large the computation, the proof can be very quickly verified on-chain. there are complex tradeoffs between the two flavors of rollups: property optimistic rollups zk rollups fixed gas cost per batch ~40,000 (a lightweight transaction that mainly just changes the value of the state root) ~500,000 (verification of a zk-snark is quite computationally intensive) withdrawal period ~1 week (withdrawals need to be delayed to give time for someone to publish a fraud proof and cancel the withdrawal if it is fraudulent) very fast (just wait for the next batch) complexity of technology low high (zk-snarks are very new and mathematically complex technology) generalizability easier (general-purpose evm rollups are already close to mainnet) harder (zk-snark proving general-purpose evm execution is much harder than proving simple computations, though there are efforts (eg. cairo) working to improve on this) per-transaction on-chain gas costs higher lower (if data in a transaction is only used to verify, and not to cause state changes, then this data can be left out, whereas in an optimistic rollup it would need to be published in case it needs to be checked in a fraud proof) off-chain computation costs lower (though there is more need for many full nodes to redo the computation) higher (zk-snark proving especially for general-purpose computation can be expensive, potentially many thousands of times more expensive than running the computation directly) in general, my own view is that in the short term, optimistic rollups are likely to win out for general-purpose evm computation and zk rollups are likely to win out for simple payments, exchange and other application-specific use cases, but in the medium to long term zk rollups will win out in all use cases as zk-snark technology improves. anatomy of a fraud proof the security of an optimistic rollup depends on the idea that if someone publishes an invalid batch into the rollup, anyone else who was keeping up with the chain and detected the fraud can publish a fraud proof, proving to the contract that that batch is invalid and should be reverted. a fraud proof claiming that a batch was invalid would contain the data in green: the batch itself (which could be checked against a hash stored on chain) and the parts of the merkle tree needed to prove just the specific accounts that were read and/or modified by the batch. the nodes in the tree in yellow can be reconstructed from the nodes in green and so do not need to be provided. this data is sufficient to execute the batch and compute the post-state root (note that this is exactly the same as how stateless clients verify individual blocks). if the computed post-state root and the provided post-state root in the batch are not the same, then the batch is fraudulent. it is guaranteed that if a batch was constructed incorrectly, and all previous batches were constructed correctly, then it is possible to create a fraud proof showing the the batch was constructed incorrectly. note the claim about previous batches: if there was more than one invalid batch published to the rollup, then it is best to try to prove the earliest one invalid. it is also, of course, guaranteed that if a batch was constructed correctly, then it is never possible to create a fraud proof showing that the batch is invalid. how does compression work? a simple ethereum transaction (to send eth) takes ~110 bytes. an eth transfer on a rollup, however, takes only ~12 bytes: parameter ethereum rollup nonce ~3 0 gasprice ~8 0-0.5 gas 3 0-0.5 to 21 4 value ~9 ~3 signature ~68 (2 + 33 + 33) ~0.5 from 0 (recovered from sig) 4 total ~112 ~12 part of this is simply superior encoding: ethereum's rlp wastes 1 byte per value on the length of each value. but there are also some very clever compression tricks that are going on: nonce: the purpose of this parameter is to prevent replays. if the current nonce of an account is 5, the next transaction from that account must have nonce 5, but once the transaction is processed the nonce in the account will be incremented to 6 so the transaction cannot be processed again. in the rollup, we can omit the nonce entirely, because we just recover the nonce from the pre-state; if someone tries replaying a transaction with an earlier nonce, the signature would fail to verify, as the signature would be checked against data that contains the new higher nonce. gasprice: we can allow users to pay with a fixed range of gasprices, eg. a choice of 16 consecutive powers of two. alternatively, we could just have a fixed fee level in each batch, or even move gas payment outside the rollup protocol entirely and have transactors pay batch creators for inclusion through a channel. gas: we could similarly restrict the total gas to a choice of consecutive powers of two. alternatively, we could just have a gas limit only at the batch level. to: we can replace the 20-byte address with an index (eg. if an address is the 4527th address added to the tree, we just use the index 4527 to refer to it. we would add a subtree to the state to store the mapping of indices to addresses). value: we can store value in scientific notation. in most cases, transfers only need 1-3 significant digits. signature: we can use bls aggregate signatures, which allows many signatures to be aggregated into a single ~32-96 byte (depending on protocol) signature. this signature can then be checked against the entire set of messages and senders in a batch all at once. the ~0.5 in the table represents the fact that there is a limit on how many signatures can be combined in an aggregate that can be verified in a single block, and so large batches would need one signature per ~100 transactions. one important compression trick that is specific to zk rollups is that if a part of a transaction is only used for verification, and is not relevant to computing the state update, then that part can be left off-chain. this cannot be done in an optimistic rollup because that data would still need to be included on-chain in case it needs to be later checked in a fraud proof, whereas in a zk rollup the snark proving correctness of the batch already proves that any data needed for verification was provided. an important example of this is privacy-preserving rollups: in an optimistic rollup the ~500 byte zk-snark used for privacy in each transaction needs to go on chain, whereas in a zk rollup the zk-snark covering the entire batch already leaves no doubt that the "inner" zk-snarks are valid. these compression tricks are key to the scalability of rollups; without them, rollups would be perhaps only a ~10x improvement on the scalability of the base chain (though there are some specific computation-heavy applications where even simple rollups are powerful), whereas with compression tricks the scaling factor can go over 100x for almost all applications. who can submit a batch? there are a number of schools of thought for who can submit a batch in an optimistic or zk rollup. generally, everyone agrees that in order to be able to submit a batch, a user must put down a large deposit; if that user ever submits a fraudulent batch (eg. with an invalid state root), that deposit would be part burned and part given as a reward to the fraud prover. but beyond that, there are many possibilities: total anarchy: anyone can submit a batch at any time. this is the simplest approach, but it has some important drawbacks. particularly, there is a risk that multiple participants will generate and attempt to submit batches in parallel, and only one of those batches can be successfully included. this leads to a large amount of wasted effort in generating proofs and/or wasted gas in publishing batches to chain. centralized sequencer: there is a single actor, the sequencer, who can submit batches (with an exception for withdrawals: the usual technique is that a user can first submit a withdrawal request, and then if the sequencer does not process that withdrawal in the next batch, then the user can submit a single-operation batch themselves). this is the most "efficient", but it is reliant on a central actor for liveness. sequencer auction: an auction is held (eg. every day) to determine who has the right to be the sequencer for the next day. this technique has the advantage that it raises funds which could be distributed by eg. a dao controlled by the rollup (see: mev auctions) random selection from pos set: anyone can deposit eth (or perhaps the rollup's own protocol token) into the rollup contract, and the sequencer of each batch is randomly selected from one of the depositors, with the probability of being selected being proportional to the amount deposited. the main drawback of this technique is that it leads to large amounts of needless capital lockup. dpos voting: there is a single sequencer selected with an auction but if they perform poorly token holders can vote to kick them out and hold a new auction to replace them. split batching and state root provision some of the rollups being currently developed are using a "split batch" paradigm, where the action of submitting a batch of layer-2 transactions and the action of submitting a state root are done separately. this has some key advantages: you can allow many sequencers in parallel to publish batches in order to improve censorship resistance, without worrying that some batches will be invalid because some other batch got included first. if a state root is fraudulent, you don't need to revert the entire batch; you can revert just the state root, and wait for someone to provide a new state root for the same batch. this gives transaction senders a better guarantee that their transactions will not be reverted. so all in all, there is a fairly complex zoo of techniques that are trying to balance between complicated tradeoffs involving efficiency, simplicity, censorship resistance and other goals. it's still too early to say which combination of these ideas works best; time will tell. how much scaling do rollups give you? on the existing ethereum chain, the gas limit is 12.5 million, and each byte of data in a transaction costs 16 gas. this means that if a block contains nothing but a single batch (we'll say a zk rollup is used, spending 500k gas on proof verification), that batch can have (12 million / 16) = 750,000 bytes of data. as shown above, a rollup for eth transfers requires only 12 bytes per user operation, meaning that the batch can contain up to 62,500 transactions. at an average block time of 13 seconds, this translates to ~4807 tps (compared to 12.5 million / 21000 / 13 ~= 45 tps for eth transfers directly on ethereum itself). here's a chart for some other example use cases: application bytes in rollup gas cost on layer 1 max scalability gain eth transfer 12 21,000 105x erc20 transfer 16 (4 more bytes to specify which token) ~50,000 187x uniswap trade ~14 (4 bytes sender + 4 bytes recipient + 3 bytes value + 1 byte max price + 1 byte misc) ~100,000 428x privacy-preserving withdrawal (optimistic rollup) 296 (4 bytes index of root + 32 bytes nullifier + 4 bytes recipient + 256 bytes zk-snark proof) ~380,000 77x privacy-preserving withdrawal (zk rollup) 40 (4 bytes index of root + 32 bytes nullifier + 4 bytes recipient) ~380,000 570x max scalability gain is calculated as (l1 gas cost) / (bytes in rollup * 16) * 12 million / 12.5 million. now, it is worth keeping in mind that these figures are overly optimistic for a few reasons. most importantly, a block would almost never just contain one batch, at the very least because there are and will be multiple rollups. second, deposits and withdrawals will continue to exist. third, in the short term usage will be low, and so fixed costs will dominate. but even with these factors taken into account, scalability gains of over 100x are expected to be the norm. now what if we want to go above ~1000-4000 tps (depending on the specific use case)? here is where eth2 data sharding comes in. the sharding proposal opens up a space of 16 mb every 12 seconds that can be filled with any data, and the system guarantees consensus on the availability of that data. this data space can be used by rollups. this ~1398k bytes per sec is a 23x improvement on the ~60 kb/sec of the existing ethereum chain, and in the longer term the data capacity is expected to grow even further. hence, rollups that use eth2 sharded data can collectively process as much as ~100k tps, and even more in the future. what are some not-yet-fully-solved challenges in rollups? while the basic concept of a rollup is now well-understood, we are quite certain that they are fundamentally feasible and secure, and multiple rollups have already been deployed to mainnet, there are still many areas of rollup design that have not been well explored, and quite a few challenges in fully bringing large parts of the ethereum ecosystem onto rollups to take advantage of their scalability. some key challenges include: user and ecosystem onboarding not many applications use rollups, rollups are unfamiliar to users, and few wallets have started integrating rollups. merchants and charities do not yet accept them for payments. cross-rollup transactions efficiently moving assets and data (eg. oracle outputs) from one rollup into another without incurring the expense of going through the base layer. auditing incentives how to maximize the chance that at least one honest node actually will be fully verifying an optimistic rollup so they can publish a fraud proof if something goes wrong? for small-scale rollups (up to a few hundred tps) this is not a significant issue and one can simply rely on altruism, but for larger-scale rollups more explicit reasoning about this is needed. exploring the design space in between plasma and rollups are there techniques that put some state-update-relevant data on chain but not all of it, and is there anything useful that could come out of that? maximizing security of pre-confirmations many rollups provide a notion of "pre-confirmation" for faster ux, where the sequencer immediately provides a promise that a transaction will be included in the next batch, and the sequencer's deposit is destroyed if they break their word. but the economy security of this scheme is limited, because of the possibility of making many promises to very many actors at the same time. can this mechanism be improved? improving speed of response to absent sequencers if the sequencer of a rollup suddenly goes offline, it would be valuable to recover from that situation maximally quickly and cheaply, either quickly and cheaply mass-exiting to a different rollup or replacing the sequencer. efficient zk-vm generating a zk-snark proof that general-purpose evm code (or some different vm that existing smart contracts can be compiled to) has been executed correctly and has a given result. conclusions rollups are a powerful new layer-2 scaling paradigm, and are expected to be a cornerstone of ethereum scaling in the short and medium-term future (and possibly long-term as well). they have seen a large amount of excitement from the ethereum community because unlike previous attempts at layer-2 scaling, they can support general-purpose evm code, allowing existing applications to easily migrate over. they do this by making a key compromise: not trying to go fully off-chain, but instead leaving a small amount of data per transaction on-chain. there are many kinds of rollups, and many choices in the design space: one can have an optimistic rollup using fraud proofs, or a zk rollup using validity proofs (aka. zk-snarks). the sequencer (the user that can publish transaction batches to chain) can be either a centralized actor, or a free-for-all, or many other choices in between. rollups are still an early-stage technology, and development is continuing rapidly, but they work and some (notably loopring, zksync and deversifi) have already been running for months. expect much more exciting work to come out of the rollup space in the years to come. dark mode toggle encapsulated vs systemic complexity in protocol design 2022 feb 28 see all posts one of the main goals of ethereum protocol design is to minimize complexity: make the protocol as simple as possible, while still making a blockchain that can do what an effective blockchain needs to do. the ethereum protocol is far from perfect at this, especially since much of it was designed in 2014-16 when we understood much less, but we nevertheless make an active effort to reduce complexity whenever possible. one of the challenges of this goal, however, is that complexity is difficult to define, and sometimes, you have to trade off between two choices that introduce different kinds of complexity and have different costs. how do we compare? one powerful intellectual tool that allows for more nuanced thinking about complexity is to draw a distinction between what we will call encapsulated complexity and systemic complexity. encapsulated complexity occurs when there is a system with sub-systems that are internally complex, but that present a simple "interface" to the outside. systemic complexity occurs when the different parts of a system can't even be cleanly separated, and have complex interactions with each other. here are a few examples. bls signatures vs schnorr signatures bls signatures and schnorr signatures are two popular types of cryptographic signature schemes that can be made with elliptic curves. bls signatures appear mathematically very simple: signing: \(\sigma = h(m) * k\) verifying: \(e([1], \sigma) \stackrel{?}{=} e(h(m), k)\) \(h\) is a hash function, \(m\) is the message, and \(k\) and \(k\) are the private and public keys. so far, so simple. however, the true complexity is hidden inside the definition of the \(e\) function: elliptic curve pairings, one of the most devilishly hard-to-understand pieces of math in all of cryptography. now, consider schnorr signatures. schnorr signatures rely only on basic elliptic curves. but the signing and verification logic is somewhat more complex: so... which type of signature is "simpler"? it depends what you care about! bls signatures have a huge amount of technical complexity, but the complexity is all buried within the definition of the \(e\) function. if you treat the \(e\) function as a black box, bls signatures are actually really easy. schnorr signatures, on the other hand, have less total complexity, but they have more pieces that could interact with the outside world in tricky ways. for example: doing a bls multi-signature (a combined signature from two keys \(k_1\) and \(k_2\)) is easy: just take \(\sigma_1 + \sigma_2\). but a schnorr multi-signature requires two rounds of interaction, and there are tricky key cancellation attacks that need to be dealt with. schnorr signatures require random number generation, bls signatures do not. elliptic curve pairings in general are a powerful "complexity sponge" in that they contain large amounts of encapsulated complexity, but enable solutions with much less systemic complexity. this is also true in the area of polynomial commitments: compare the simplicity of kzg commitments (which require pairings) to the much more complicated internal logic of inner product arguments (which do not). cryptography vs cryptoeconomics one important design choice that appears in many blockchain designs is that of cryptography versus cryptoeconomics. often (eg. in rollups) this comes in the form of a choice between validity proofs (aka. zk-snarks) and fraud proofs. zk-snarks are complex technology. while the basic ideas behind how they work can be explained in a single post, actually implementing a zk-snark to verify some computation involves many times more complexity than the computation itself (hence why zk-snarks for the evm are still under development while fraud proofs for the evm are already in the testing stage). implementing a zk-snark effectively involves circuit design with special-purpose optimization, working with unfamiliar programming languages, and many other challenges. fraud proofs, on the other hand, are inherently simple: if someone makes a challenge, you just directly run the computation on-chain. for efficiency, a binary-search scheme is sometimes added, but even that doesn't add too much complexity. but while zk-snarks are complex, their complexity is encapsulated complexity. the relatively light complexity of fraud proofs, on the other hand, is systemic. here are some examples of systemic complexity that fraud proofs introduce: they require careful incentive engineering to avoid the verifier's dilemma. if done in-consensus, they require extra transaction types for the fraud proofs, along with reasoning about what happens if many actors compete to submit a fraud proof at the same time. they depend on a synchronous network. they allow censorship attacks to be also used to commit theft. rollups based on fraud proofs require liquidity providers to support instant withdrawals. for these reasons, even from a complexity perspective purely cryptographic solutions based on zk-snarks are likely to be long-run safer: zk-snarks have are more complicated parts that some people have to think about, but they have fewer dangling caveats that everyone has to think about. miscellaneous examples proof of work (nakamoto consensus) low encapsulated complexity, as the mechanism is extremely simple and easy to understand, but higher systemic complexity (eg. selfish mining attacks). hash functions high encapsulated complexity, but very easy-to-understand properties so low systemic complexity. random shuffling algorithms shuffling algorithms can either be internally complicated (as in whisk) but lead to easy-to-understand guarantees of strong randomness, or internally simpler but lead to randomness properties that are weaker and more difficult to analyze (systemic complexity). miner extractable value (mev) a protocol that is powerful enough to support complex transactions can be fairly simple internally, but those complex transactions can have complex systemic effects on the protocol's incentives by contributing to the incentive to propose blocks in very irregular ways. verkle trees verkle trees do have some encapsulated complexity, in fact quite a bit more than plain merkle hash trees. systemically, however, verkle trees present the exact same relatively clean-and-simple interface of a key-value map. the main systemic complexity "leak" is the possibility of an attacker manipulating the tree to make a particular value have a very long branch; but this risk is the same for both verkle trees and merkle trees. how do we make the tradeoff? often, the choice with less encapsulated complexity is also the choice with less systemic complexity, and so there is one choice that is obviously simpler. but at other times, you have to make a hard choice between one type of complexity and the other. what should be clear at this point is that complexity is less dangerous if it is encapsulated. the risks from complexity of a system are not a simple function of how long the specification is; a small 10-line piece of the specification that interacts with every other piece adds more complexity than a 100-line function that is otherwise treated as a black box. however, there are limits to this approach of preferring encapsulated complexity. software bugs can occur in any piece of code, and as it gets bigger the probability of a bug approaches 1. sometimes, when you need to interact with a sub-system in an unexpected and new way, complexity that was originally encapsulated can become systemic. one example of the latter is ethereum's current two-level state tree, which features a tree of account objects, where each account object in turn has its own storage tree. this tree structure is complex, but at the beginning the complexity seemed to be well-encapsulated: the rest of the protocol interacts with the tree as a key/value store that you can read and write to, so we don't have to worry about how the tree is structured. later, however, the complexity turned out to have systemic effects: the ability of accounts to have arbitrarily large storage trees meant that there was no way to reliably expect a particular slice of the state (eg. "all accounts starting with 0x1234") to have a predictable size. this makes it harder to split up the state into pieces, complicating the design of syncing protocols and attempts to distribute the storage process. why did encapsulated complexity become systemic? because the interface changed. the fix? the current proposal to move to verkle trees also includes a move to a well-balanced single-layer design for the tree, ultimately, which type of complexity to favor in any given situation is a question with no easy answers. the best that we can do is to have an attitude of moderately favoring encapsulated complexity, but not too much, and exercise our judgement in each specific case. sometimes, a sacrifice of a little bit of systemic complexity to allow a great reduction of encapsulated complexity really is the best thing to do. and other times, you can even misjudge what is encapsulated and what isn't. each situation is different. gav’s ethereum ðξv update v | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search gav’s ethereum ðξv update v posted by gavin wood on march 2, 2015 research & development i was woken by vitalik's call at 5:55 this morning; pitch black outside, nighttime was still upon us. nonetheless, it was time to leave and this week had best start on the right foot. the 25-minute walk in darkness from the zug-based headquarters to the train station was wet. streetlights reflecting off the puddles on the clean swiss streets provided a picturesque, if quiet, march into town. i couldn't help but think the rain running down my face was a very liquid reminder of the impending seasonal change, and then, on consideration, how fast the last nine months had gone. solid foundations the last week was spent in zug by the ethereum foundation board and ðξv leadership: vitalik, mihai and taylor who officially form the founation's board, anthony and joseph as the other official advisors and aeron & jutta as the ðξv executive joined by jeff and myself wearing multiple hats of ðξv and advisory). the chief outcome of this was the dissemination of vitalik's superb plan to reform the foundation and turn it into a professional entity. the board will be recruited from accomplished professionals with minimal conflicts of interest; the present set of "founders" officially retired from those positions and a professional executive recruited, the latter process lead by joseph. anthony will take a greater ambassadorial role for ethereum in china and north america. conversely, ðξv will function much more as a department of the foundation's executive rather than a largely independent entity. finally, i presented the release strategy to the others; an event after which i've never seen quite so many photos taken of a whiteboard. needless to say, all was well received by the board and advisors. more information will be coming soon. as i write this, i'm sitting on a crowded early commuter train, vinay gupta in tow, who recently took on a much more substantive role this week as release coordinator. he'll be helping with release strategy and to keep you informed of our release process. this week, which might rather dramatically be described as 'pivotal' in the release process, will see jeff, vitalk and me sit around a table and develop all the poc-9 changes, related unit tests, and integrations in three days, joined by our indomitable master of testing, christoph. the outcome of this week will inform our announcement which will come later this week outlining in clear terms what we will be releasing and when. i'm sorry it has been so long without an update. the last 2 months has been somewhat busy, choked up with travel and meetings, with the remaining time soaked up by coding, team-leading and management. the team is now substantially formed; the formal security audit started four weeks ago; the bounty programme is running smoothly. the latter processes are the exceedingly capable hands of jutta and gustav. aeron, meanwhile will be stepping down as the ðξv head of finance and operations and assuming the role he was initially brought aboard for, system modelling. we'll hopefully be able to announce his successors next week (yes, that was plural; he has been doing the jobs of 2.5 people over the last few months). we are also in the process of forming partnerships with third parties in the industry; george, jutta and myself managing this process; i'm happy to announce that at least three exchanges will be supporting ether from day one on their trading platforms (details of which we'll annouce soon), with more exchanges to follow. marek and alex are providing technical supprt there with marek going so far as to make a substantial reference exchange implementation. i also finished the first draft of icap, the ethereum inter-exchange client address protocol, an iban-compatible system for referencing and transacting to client accounts aimed to streamline the process of transfering funds, worry-free between exchanges and, ultimately, make kyc and aml pains a thing of the past. the iban compatibility may even provide possibility of easy integration with existing banking infrastructure in some future. developments proof-of-concept releases vii and viii were released. natspec, "natural language specification format" and the basis of our transaction security was prototyped and integrated. under marek's watch, now helped by fabian, ethereum.js is truly coming of age with a near source-level compatibility with solidity on contract interaction and support for the typed abi with calling and events, the latter providing hassle-free state-change reporting. mix, our ide, underwent its first release and after some teethng issues is getting good use thanks to the excellent work done by arkadiy and yann. solidity had numerous features added and is swiftly approaching 1.0 status with christian, lefteris and liana to thank. marian's work goes ever forward on the network monitoring system while sven and heiko have been working diligently on the stress testing infrastructure which analyses and tests the peer network formation and performance. they'll soon be joined by alex and lefteris to accellerate this programme. so one of the major things that needed sorting for the next release is the proof-of-work algorithm that we'll use. this had a number of requirements, two of which were actually pulling in opposite directions, but basically it had to be light-client-friendly algorithm whose speed-of-mining is proportional to the io-bandwidth and which requires a considerable amount of ram to do so. there was a vague consensus that we (well.. vitalik and matthew) head in the direction of a hasimoto-like algorithm (a proof-of-work designed for the bitcoin blockchain that aims to be io-bound, meaning, roughly, that to make it go any faster, you'd need to add more memory rather than just sponsoring a smaller/faster asic). since our blockchain has a number of important differences with the bitcoin blockchain (mainly in transaction density), stemming from the extremely short 12s block time we're aiming for, we would have to use not the blockchain data itself like hashimoto but rather an artifcially created dataset, done with an algorithm known as dagger (yes, some will remember it as vitalik's first and flawed attempt at a memory-hard proof-of-work). while this looked like a good direction to be going in, a swift audit of vitalik and matt's initial algorithm by tim hughes (ex-director of technology at frontier developments and expert in low-level cpu and gpu operation and optimisation) showed major flaws. with his help, they were able to work together to devise a substantially more watertight algorithm that, we are confident to say, should make the job of developing an fpga/asic sufficiently difficult, especially given our determination to switch to a proof-of-stake system within the next 6-12 months. last, but not least, the new website was launched. kudos to ian and konstantin for mucking down and getting it done. next stop will be the developer site, which will be loosely based on the excellent resource at qt.io, the aim to provide a one-stop extravaganza of up to date reference documentation, curated tutorials, examples, recipes, downloads, issue tracking, and build status. onwards so, as alex, our networking maestro might say, these are exciting times. when deep in nitty gritty of development you sometimes forget quite how world-altering the technology you're creating is, which is probably just as well since the gravity of the matter at hand would be continually distracting. nonetheless, when one starts considering the near-term alterations that we can really bring one realises that the wave of change is at once unavoidable and heading straight for you. for what it's worth, i find an excellent accompaniment to this crazy life is the superb music of pretty lights. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the p + epsilon attack | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the p + epsilon attack posted by vitalik buterin on january 28, 2015 research & development special thanks to andrew miller for coming up with this attack, and to zack hess, vlad zamfir and paul sztorc for discussion and responses one of the more interesting surprises in cryptoeconomics in recent weeks came from an attack on schellingcoin conceived by andrew miller earlier this month. although it has always been understood that schellingcoin, and similar systems (including the more advanced truthcoin consensus), rely on what is so far a new and untested cryptoeconomic security assumption that one can safely rely on people acting honestly in a simultaneous consensus game just because they believe that everyone else will the problems that have been raised so far have to do with relatively marginal issues like an attacker's ability to exert small but increasing amounts of influence on the output over time by applying continued pressure. this attack, on the other hand, shows a much more fundamental problem. the scenario is described as follows. suppose that there exists a simple schelling game where users vote on whether or not some particular fact is true (1) or false (0); say in our example that it's actually false. each user can either vote 1 or 0. if a user votes the same as the majority, they get a reward of p; otherwise they get 0. thus, the payoff matrix looks as follows: you vote 0 you vote 1 others vote 0 p 0 others vote 1 0 p the theory is that if everyone expects everyone else to vote truthfully, then their incentive is to also vote truthfully in order to comply with the majority, and that's the reason why one can expect others to vote truthfully in the first place; a self-reinforcing nash equilibrium. now, the attack. suppose that the attacker credibly commits (eg. via an ethereum contract, by simply putting one's reputation at stake, or by leveraging the reputation of a trusted escrow provider) to pay out x to voters who voted 1 after the game is over, where x = p + ε if the majority votes 0, and x = 0 if the majority votes 1. now, the payoff matrix looks like this: you vote 0 you vote 1 others vote 0 p p + ε others vote 1 0 p thus, it's a dominant strategy for anyone to vote 1 no matter what you think the majority will do. hence, assuming the system is not dominated by altruists, the majority will vote 1, and so the attacker will not need to pay anything at all. the attack has successfully managed to take over the mechanism at zero cost. note that this differs from nicholas houy's argument about zero-cost 51% attacks on proof of stake (an argument technically extensible to asic-based proof of work) in that here no epistemic takeover is required; even if everyone remains dead set in a conviction that the attacker is going to fail, their incentive is still to vote to support the attacker, because the attacker takes on the failure risk themselves. salvaging schelling schemes there are a few avenues that one can take to try to salvage the schelling mechanism. one approach is that instead of round n of the schelling consensus itself deciding who gets rewarded based on the "majority is right" principle, we use round n + 1 to determine who should be rewarded during round n, with the default equilibrium being that only people who voted correctly during round n (both on the actual fact in question and on who should be rewarded in round n 1) should be rewarded. theoretically, this requires an attacker wishing to perform a cost-free attack to corrupt not just one round, but also all future rounds, making the required capital deposit that the attacker must make unbounded. however, this approach has two flaws. first, the mechanism is fragile: if the attacker manages to corrupt some round in the far future by actually paying up p + ε to everyone, regardless of who wins, then the expectation of that corrupted round causes an incentive to cooperate with the attacker to back-propagate to all previous rounds. hence, corrupting one round is costly, but corrupting thousands of rounds is not much more costly. second, because of discounting, the required deposit to overcome the scheme does not need to be infinite; it just needs to be very very large (ie. inversely proportional to the prevailing interest rate). but if all we want is to make the minimum required bribe larger, then there exists a much simpler and better strategy for doing so, pioneered by paul storcz: require participants to put down a large deposit, and build in a mechanism by which the more contention there is, the more funds are at stake. at the limit, where slightly over 50% of votes are in favor of one outcome and 50% in favor of the other, the entire deposit it taken away from minority voters. this ensures that the attack still works, but the bribe must now be greater than the deposit (roughly equal to the payout divided by the discounting rate, giving us equal performance to the infinite-round game) rather than just the payout for each round. hence, in order to overcome such a mechanism, one would need to be able to prove that one is capable of pulling off a 51% attack, and perhaps we may simply be comfortable with assuming that attackers of that size do not exist. another approach is to rely on counter-coordination; essentially, somehow coordinate, perhaps via credible commitments, on voting a (if a is the truth) with probability 0.6 and b with probability 0.4, the theory being that this will allow users to (probabilistically) claim the mechanism's reward and a portion of the attacker's bribe at the same time. this (seems to) work particularly well in games where instead of paying out a constant reward to each majority-compliant voter, the game is structured to have a constant total payoff, adjusting individual payoffs to accomplish this goal is needed. in such situations, from a collective-rationality standpoint it is indeed the case that the group earns a highest profit by having 49% of its members vote b to claim the attacker's reward and 51% vote a to make sure the attacker's reward is paid out. however, this approach itself suffers from the flaw that, if the attacker's bribe is high enough, even from there one can defect. the fundamental problem is that given a probabilistic mixed strategy between a and b, for each the return always changes (almost) linearly with the probability parameter. hence, if, for the individual, it makes more sense to vote for b than for a, it will also make more sense to vote with probability 0.51 for b than with probability 0.49 for b, and voting with probability 1 for b will work even better. hence, everyone will defect from the "49% for 1" strategy by simply always voting for 1, and so 1 will win and the attacker will have succeeded in the costless takeover. the fact that such complicated schemes exist, and come so close to "seeming to work" suggests that perhaps in the near future some complex counter-coordination scheme will emerge that actually does work; however, we must be prepared for the eventuality that no such scheme will be developed. further consequences given the sheer number of cryptoeconomic mechanisms that schellingcoin makes possible, and the importance of such schemes in nearly all purely "trust-free" attempts to forge any kind of link between the cryptographic world and the real world, this attack poses a potential serious threat although, as we will later see, schelling schemes as a category are ultimately partially salvageable. however, what is more interesting is the much larger class of mechanisms that don't look quite like schellingcoin at first glance, but in fact have very similar sets of strengths and weaknesses. particularly, let us point to one very specific example: proof of work. proof of work is in fact a multi-equilibrium game in much the same way that schelling schemes are: if there exist two forks, a and b, then if you mine on the fork that ends up winning you get 25 btc and if you mine on the fork that ends up losing you get nothing. you mine on a you mine on b others mine on a 25 0 others mine on b 0 25 now, suppose that an attacker launches a double-spend attack against many parties simultaneously (this requirement ensures that there is no single party with very strong incentive to oppose the attacker, opposition instead becoming a public good; alternatively the double spend could be purely an attempt to crash the price with the attacker shorting at 10x leverage), and call the "main" chain a and the attacker's new double-spend fork b. by default, everyone expects a to win. however, the attacker credibly commits to paying out 25.01 btc to everyone who mines on b if b ends up losing. hence, the payoff matrix becomes: you mine on a you mine on b others mine on a 25 25.01 others mine on b 0 25 thus, mining on b is a dominant strategy regardless of one's epistemic beliefs, and so everyone mines on b, and so the attacker wins and pays out nothing at all. particularly, note that in proof of work we do not have deposits, so the level of bribe required is proportional only to the mining reward multiplied by the fork length, not the capital cost of 51% of all mining equipment. hence, from a cryptoeconomic security standpoint, one can in some sense say that proof of work has virtually no cryptoeconomic security margin at all (if you are tired of opponents of proof of stake pointing you to this article by andrew poelstra, feel free to link them here in response). if one is genuinely uncomfortable with the weak subjectivity condition of pure proof of stake, then it follows that the correct solution may perhaps be to augment proof of work with hybrid proof of stake by adding security deposits and double-voting-penalties to mining. of course, in practice, proof of work has survived despite this flaw, and indeed it may continue to survive for a long time still; it may just be the case that there's a high enough degree of altruism that attackers are not actually 100% convinced that they will succeed but then, if we are allowed to rely on altruism, naive proof of stake works fine too. hence, schelling schemes too may well simply end up working in practice, even if they are not perfectly sound in theory. the next part of this post will discuss the concept of "subjective" mechanisms in more detail, and how they can be used to theoretically get around some of these problems. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle verkle trees 2021 jun 18 see all posts special thanks to dankrad feist and justin drake for feedback and review. verkle trees are shaping up to be an important part of ethereum's upcoming scaling upgrades. they serve the same function as merkle trees: you can put a large amount of data into a verkle tree, and make a short proof ("witness") of any single piece, or set of pieces, of that data that can be verified by someone who only has the root of the tree. the key property that verkle trees provide, however, is that they are much more efficient in proof size. if a tree contains a billion pieces of data, making a proof in a traditional binary merkle tree would require about 1 kilobyte, but in a verkle tree the proof would be less than 150 bytes a reduction sufficient to make stateless clients finally viable in practice. verkle trees are still a new idea; they were first introduced by john kuszmaul in this paper from 2018, and they are still not as widely known as many other important new cryptographic constructions. this post will explain what verkle trees are and how the cryptographic magic behind them works. the price of their short proof size is a higher level of dependence on more complicated cryptography. that said, the cryptography still much simpler, in my opinion, than the advanced cryptography found in modern zk snark schemes. in this post i'll do the best job that i can at explaining it. merkle patricia vs verkle tree node structure in terms of the structure of the tree (how the nodes in the tree are arranged and what they contain), a verkle tree is very similar to the merkle patricia tree currently used in ethereum. every node is either (i) empty, (ii) a leaf node containing a key and value, or (iii) an intermediate node that has some fixed number of children (the "width" of the tree). the value of an intermediate node is computed as a hash of the values of its children. the location of a value in the tree is based on its key: in the diagram below, to get to the node with key 4cc, you start at the root, then go down to the child at position 4, then go down to the child at position c (remember: c = 12 in hexadecimal), and then go down again to the child at position c. to get to the node with key baaa, you go to the position-b child of the root, and then the position-a child of that node. the node at path (b,a) directly contains the node with key baaa, because there are no other keys in the tree starting with ba. the structure of nodes in a hexary (16 children per parent) verkle tree, here filled with six (key, value) pairs. the only real difference in the structure of verkle trees and merkle patricia trees is that verkle trees are wider in practice. much wider. patricia trees are at their most efficient when width = 2 (so ethereum's hexary patricia tree is actually quite suboptimal). verkle trees, on the other hand, get shorter and shorter proofs the higher the width; the only limit is that if width gets too high, proofs start to take too long to create. the verkle tree proposed for ethereum has a width of 256, and some even favor raising it to 1024 (!!). commitments and proofs in a merkle tree (including merkle patricia trees), the proof of a value consists of the entire set of sister nodes: the proof must contain all nodes in the tree that share a parent with any of the nodes in the path going down to the node you are trying to prove. that may be a little complicated to understand, so here's a picture of a proof for the value in the 4ce position. sister nodes that must be included in the proof are highlighted in red. that's a lot of nodes! you need to provide the sister nodes at each level, because you need the entire set of children of a node to compute the value of that node, and you need to keep doing this until you get to the root. you might think that this is not that bad because most of the nodes are zeroes, but that's only because this tree has very few nodes. if this tree had 256 randomly-allocated nodes, the top layer would almost certainly have all 16 nodes full, and the second layer would on average be ~63.3% full. in a verkle tree, on the other hand, you do not need to provide sister nodes; instead, you just provide the path, with a little bit extra as a proof. this is why verkle trees benefit from greater width and merkle patricia trees do not: a tree with greater width leads to shorter paths in both cases, but in a merkle patricia tree this effect is overwhelmed by the higher cost of needing to provide all the width 1 sister nodes per level in a proof. in a verkle tree, that cost does not exist. so what is this little extra that we need as a proof? to understand that, we first need to circle back to one key detail: the hash function used to compute an inner node from its children is not a regular hash. instead, it's a vector commitment. a vector commitment scheme is a special type of hash function, hashing a list \(h(z_1, z_2 ... z_n) \rightarrow c\). but vector commitments have the special property that for a commitment \(c\) and a value \(z_i\), it's possible to make a short proof that \(c\) is the commitment to some list where the value at the i'th position is \(z_i\). in a verkle proof, this short proof replaces the function of the sister nodes in a merkle patricia proof, giving the verifier confidence that a child node really is the child at the given position of its parent node. no sister nodes required in a proof of a value in the tree; just the path itself plus a few short proofs to link each commitment in the path to the next. in practice, we use a primitive even more powerful than a vector commitment, called a polynomial commitment. polynomial commitments let you hash a polynomial, and make a proof for the evaluation of the hashed polynomial at any point. you can use polynomial commitments as vector commitments: if we agree on a set of standardized coordinates \((c_1, c_2 ... c_n)\), given a list \((y_1, y_2 ... y_n)\) you can commit to the polynomial \(p\) where \(p(c_i) = y_i\) for all \(i \in [1..n]\) (you can find this polynomial with lagrange interpolation). i talk about polynomial commitments at length in my article on zk-snarks. the two polynomial commitment schemes that are the easiest to use are kzg commitments and bulletproof-style commitments (in both cases, a commitment is a single 32-48 byte elliptic curve point). polynomial commitments give us more flexibility that lets us improve efficiency, and it just so happens that the simplest and most efficient vector commitments available are the polynomial commitments. this scheme is already very powerful as it is: if you use a kzg commitment and proof, the proof size is 96 bytes per intermediate node, nearly 3x more space-efficient than a simple merkle proof if we set width = 256. however, it turns out that we can increase space-efficiency even further. merging the proofs instead of requiring one proof for each commitment along the path, by using the extra properties of polynomial commitments we can make a single fixed-size proof that proves all parent-child links between commitments along the paths for an unlimited number of keys. we do this using a scheme that implements multiproofs through random evaluation. but to use this scheme, we first need to convert the problem into a more structured one. we have a proof of one or more values in a verkle tree. the main part of this proof consists of the intermediary nodes along the path to each node. for each node that we provide, we also have to prove that it actually is the child of the node above it (and in the correct position). in our single-value-proof example above, we needed proofs to prove: that the key: 4ce node actually is the position-e child of the prefix: 4c intermediate node. that the prefix: 4c intermediate node actually is the position-c child of the prefix: 4 intermediate node. that the prefix: 4 intermediate node actually is the position-4 child of the root if we had a proof proving multiple values (eg. both 4ce and 420), we would have even more nodes and even more linkages. but in any case, what we are proving is a sequence of statements of the form "node a actually is the position-i child of node b". if we are using polynomial commitments, this turns into equations: \(a(x_i) = y\), where \(y\) is the hash of the commitment to \(b\). the details of this proof are technical and better explained by dankrad feist than myself. by far the bulkiest and time-consuming step in the proof generation involves computing a polynomial \(g\) of the form: \(g(x) = r^0\frac{a_0(x) y_0}{x x_0} + r^1\frac{a_1(x) y_1}{x x_1} + ... + r^n\frac{a_n(x) y_n}{x x_n}\) it is only possible to compute each term \(r^i\frac{a_i(x) y_i}{x x_i}\) if that expression is a polynomial (and not a fraction). and that requires \(a_i(x)\) to equal \(y_i\) at the point \(x_i\). we can see this with an example. suppose: \(a_i(x) = x^2 + x + 3\) we are proving for \((x_i = 2, y_i = 9)\). \(a_i(2)\) does equal \(9\) so this will work. \(a_i(x) 9 = x^2 + x 6\), and \(\frac{x^2 + x 6}{x 2}\) gives a clean \(x 3\). but if we tried to fit in \((x_i = 2, y_i = 10)\), this would not work; \(x^2 + x 7\) cannot be cleanly divided by \(x 2\) without a fractional remainder. the rest of the proof involves providing a polynomial commitment to \(g(x)\) and then proving that the commitment is actually correct. once again, see dankrad's more technical description for the rest of the proof. one single proof proves an unlimited number of parent-child relationships. and there we have it, that's what a maximally efficient verkle proof looks like. key properties of proof sizes using this scheme dankrad's multi-random-evaluation proof allows the prover to prove an arbitrary number of evaluations \(a_i(x_i) = y_i\), given commitments to each \(a_i\) and the values that are being proven. this proof is constant size (one polynomial commitment, one number, and two proofs; 128-1000 bytes depending on what scheme is being used). the \(y_i\) values do not need to be provided explicitly, as they can be directly computed from the other values in the verkle proof: each \(y_i\) is itself the hash of the next value in the path (either a commitment or a leaf). the \(x_i\) values also do not need to be provided explicitly, since the paths (and hence the \(x_i\) values) can be computed from the keys and the coordinates derived from the paths. hence, all we need is the leaves (keys and values) that we are proving, as well as the commitments along the path from each leaf to the root. assuming a width-256 tree, and \(2^{32}\) nodes, a proof would require the keys and values that are being proven, plus (on average) three commitments for each value along the path from that value to the root. if we are proving many values, there are further savings: no matter how many values you are proving, you will not need to provide more than the 256 values at the top level. proof sizes (bytes). rows: tree size, cols: key/value pairs proven 1 10 100 1,000 10,000 256 176 176 176 176 176 65,536 224 608 4,112 12,176 12,464 16,777,216 272 1,040 8,864 59,792 457,616 4,294,967,296 320 1,472 13,616 107,744 937,472 assuming width 256, and 48-byte kzg commitments/proofs. note also that this assumes a maximally even tree; for a realistic randomized tree, add a depth of ~0.6 (so ~30 bytes per element). if bulletproof-style commitments are used instead of kzg, it's safe to go down to 32 bytes, so these sizes can be reduced by 1/3. prover and verifier computation load the bulk of the cost of generating a proof is computing each \(r^i\frac{a_i(x) y_i}{x x_i}\) expression. this requires roughly four field operations (ie. 256 bit modular arithmetic operations) times the width of the tree. this is the main constraint limiting verkle tree widths. fortunately, four field operations is a small cost: a single elliptic curve multiplication typically takes hundreds of field operations. hence, verkle tree widths can go quite high; width 256-1024 seems like an optimal range. to edit the tree, we need to "walk up the tree" from the leaf to the root, changing the intermediate commitment at each step to reflect the change that happened lower down. fortunately, we don't have to re-compute each commitment from scratch. instead, we take advantage of the homomorphic property: given a polynomial commitment \(c = com(f)\), we can compute \(c' = com(f + g)\) by taking \(c' = c + com(g)\). in our case, \(g = l_i * (v_{new} v_{old})\), where \(l_i\) is a pre-computed commitment for the polynomial that equals 1 at the position we're trying to change and 0 everywhere else. hence, a single edit requires ~4 elliptic curve multiplications (one per commitment between the leaf and the root, this time including the root), though these can be sped up considerably by pre-computing and storing many multiples of each \(l_i\). proof verification is quite efficient. for a proof of n values, the verifier needs to do the following steps, all of which can be done within a hundred milliseconds for even thousands of values: one size-\(n\) elliptic curve fast linear combination about \(4n\) field operations (ie. 256 bit modular arithmetic operations) a small constant amount of work that does not depend on the size of the proof note also that, like merkle patricia proofs, a verkle proof gives the verifier enough information to modify the values in the tree that are being proven and compute the new root hash after the changes are applied. this is critical for verifying that eg. state changes in a block were processed correctly. conclusions verkle trees are a powerful upgrade to merkle proofs that allow for much smaller proof sizes. instead of needing to provide all "sister nodes" at each level, the prover need only provide a single proof that proves all parent-child relationships between all commitments along the paths from each leaf node to the root. this allows proof sizes to decrease by a factor of ~6-8 compared to ideal merkle trees, and by a factor of over 20-30 compared to the hexary patricia trees that ethereum uses today (!!). they do require more complex cryptography to implement, but they present the opportunity for large gains to scalability. in the medium term, snarks can improve things further: we can either snark the already-efficient verkle proof verifier to reduce witness size to near-zero, or switch back to snarked merkle proofs if/when snarks get much better (eg. through gkr, or very-snark-friendly hash functions, or asics). further down the line, the rise of quantum computing will force a change to starked merkle proofs with hashes as it makes the linear homomorphisms that verkle trees depend on insecure. but for now, they give us the same scaling gains that we would get with such more advanced technologies, and we already have all the tools that we need to implement them efficiently. dark mode toggle the different types of zk-evms 2022 aug 04 see all posts special thanks to the pse, polygon hermez, zksync, scroll, matter labs and starkware teams for discussion and review. there have been many "zk-evm" projects making flashy announcements recently. polygon open-sourced their zk-evm project, zksync released their plans for zksync 2.0, and the relative newcomer scroll announced their zk-evm recently. there is also the ongoing effort from the privacy and scaling explorations team, nicolas liochon et al's team, an alpha compiler from the evm to starkware's zk-friendly language cairo, and certainly at least a few others i have missed. the core goal of all of these projects is the same: to use zk-snark technology to make cryptographic proofs of execution of ethereum-like transactions, either to make it much easier to verify the ethereum chain itself or to build zk-rollups that are (close to) equivalent to what ethereum provides but are much more scalable. but there are subtle differences between these projects, and what tradeoffs they are making between practicality and speed. this post will attempt to describe a taxonomy of different "types" of evm equivalence, and what are the benefits and costs of trying to achieve each type. overview (in chart form) type 1 (fully ethereum-equivalent) type 1 zk-evms strive to be fully and uncompromisingly ethereum-equivalent. they do not change any part of the ethereum system to make it easier to generate proofs. they do not replace hashes, state trees, transaction trees, precompiles or any other in-consensus logic, no matter how peripheral. advantage: perfect compatibility the goal is to be able to verify ethereum blocks as they are today or at least, verify the execution-layer side (so, beacon chain consensus logic is not included, but all the transaction execution and smart contract and account logic is included). type 1 zk-evms are what we ultimately need make the ethereum layer 1 itself more scalable. in the long term, modifications to ethereum tested out in type 2 or type 3 zk-evms might be introduced into ethereum proper, but such a re-architecting comes with its own complexities. type 1 zk-evms are also ideal for rollups, because they allow rollups to re-use a lot of infrastructure. for example, ethereum execution clients can be used as-is to generate and process rollup blocks (or at least, they can be once withdrawals are implemented and that functionality can be re-used to support eth being deposited into the rollup), so tooling such as block explorers, block production, etc is very easy to re-use. disadvantage: prover time ethereum was not originally designed around zk-friendliness, so there are many parts of the ethereum protocol that take a large amount of computation to zk-prove. type 1 aims to replicate ethereum exactly, and so it has no way of mitigating these inefficiencies. at present, proofs for ethereum blocks take many hours to produce. this can be mitigated either by clever engineering to massively parallelize the prover or in the longer term by zk-snark asics. who's building it? the zk-evm community edition (bootstrapped by community contributors including privacy and scaling explorations, the scroll team, taiko and others) is a tier 1 zk-evm. type 2 (fully evm-equivalent) type 2 zk-evms strive to be exactly evm-equivalent, but not quite ethereum-equivalent. that is, they look exactly like ethereum "from within", but they have some differences on the outside, particularly in data structures like the block structure and state tree. the goal is to be fully compatible with existing applications, but make some minor modifications to ethereum to make development easier and to make proof generation faster. advantage: perfect equivalence at the vm level type 2 zk-evms make changes to data structures that hold things like the ethereum state. fortunately, these are structures that the evm itself cannot access directly, and so applications that work on ethereum would almost always still work on a type 2 zk-evm rollup. you would not be able to use ethereum execution clients as-is, but you could use them with some modifications, and you would still be able to use evm debugging tools and most other developer infrastructure. there are a small number of exceptions. one incompatibility arises for applications that verify merkle proofs of historical ethereum blocks to verify claims about historical transactions, receipts or state (eg. bridges sometimes do this). a zk-evm that replaces keccak with a different hash function would break these proofs. however, i usually recommend against building applications this way anyway, because future ethereum changes (eg. verkle trees) will break such applications even on ethereum itself. a better alternative would be for ethereum itself to add future-proof history access precompiles. disadvantage: improved but still slow prover time type 2 zk-evms provide faster prover times than type 1 mainly by removing parts of the ethereum stack that rely on needlessly complicated and zk-unfriendly cryptography. particularly, they might change ethereum's keccak and rlp-based merkle patricia tree and perhaps the block and receipt structures. type 2 zk-evms might instead use a different hash function, eg. poseidon. another natural modification is modifying the state tree to store the code hash and keccak, removing the need to verify hashes to process the extcodehash and extcodecopy opcodes. these modifications significantly improve prover times, but they do not solve every problem. the slowness from having to prove the evm as-is, with all of the inefficiencies and zk-unfriendliness inherent to the evm, still remains. one simple example of this is memory: because an mload can read any 32 bytes, including "unaligned" chunks (where the start and end are not multiples of 32), an mload can't simply be interpreted as reading one chunk; rather, it might require reading two consecutive chunks and performing bit operations to combine the result. who's building it? scroll's zk-evm project is building toward a type 2 zk-evm, as is polygon hermez. that said, neither project is quite there yet; in particular, a lot of the more complicated precompiles have not yet been implemented. hence, at the moment both projects are better considered type 3. type 2.5 (evm-equivalent, except for gas costs) one way to significantly improve worst-case prover times is to greatly increase the gas costs of specific operations in the evm that are very difficult to zk-prove. this might involve precompiles, the keccak opcode, and possibly specific patterns of calling contracts or accessing memory or storage or reverting. changing gas costs may reduce developer tooling compatibility and break a few applications, but it's generally considered less risky than "deeper" evm changes. developers should take care to not require more gas in a transaction than fits into a block, to never make calls with hard-coded amounts of gas (this has already been standard advice for developers for a long time). an alternative way to manage resource constraints is to simply set hard limits on the number of times each operation can be called. this is easier to implement in circuits, but plays much less nicely with evm security assumptions. i would call this approach type 3 rather than type 2.5. type 3 (almost evm-equivalent) type 3 zk-evms are almost evm-equivalent, but make a few sacrifices to exact equivalence to further improve prover times and make the evm easier to develop. advantage: easier to build, and faster prover times type 3 zk-evms might remove a few features that are exceptionally hard to implement in a zk-evm implementation. precompiles are often at the top of the list here;. additionally, type 3 zk-evms sometimes also have minor differences in how they treat contract code, memory or stack. disadvantage: more incompatibility the goal of a type 3 zk-evm is to be compatible with most applications, and require only minimal re-writing for the rest. that said, there will be some applications that would need to be rewritten either because they use pre-compiles that the type 3 zk-evm removes or because of subtle dependencies on edge cases that the vms treat differently. who's building it? scroll and polygon are both type 3 in their current forms, though they're expected to improve compatibility over time. polygon has a unique design where they are zk-verifying their own internal language called zkasm, and they interpret zk-evm code using the zkasm implementation. despite this implementation detail, i would still call this a genuine type 3 zk-evm; it can still verify evm code, it just uses some different internal logic to do it. today, no zk-evm team wants to be a type 3; type 3 is simply a transitional stage until the complicated work of adding precompiles is finished and the project can move to type 2.5. in the future, however, type 1 or type 2 zk-evms may become type 3 zk-evms voluntarily, by adding in new zk-snark-friendly precompiles that provide functionality for developers with low prover times and gas costs. type 4 (high-level-language equivalent) a type 4 system works by taking smart contract source code written in a high-level language (eg. solidity, vyper, or some intermediate that both compile to) and compiling that to some language that is explicitly designed to be zk-snark-friendly. advantage: very fast prover times there is a lot of overhead that you can avoid by not zk-proving all the different parts of each evm execution step, and starting from the higher-level code directly. i'm only describing this advantage with one sentence in this post (compared to a big bullet point list below for compatibility-related disadvantages), but that should not be interpreted as a value judgement! compiling from high-level languages directly really can greatly reduce costs and help decentralization by making it easier to be a prover. disadvantage: more incompatibility a "normal" application written in vyper or solidity can be compiled down and it would "just work", but there are some important ways in which very many applications are not "normal": contracts may not have the same addresses in a type 4 system as they do in the evm, because create2 contract addresses depend on the exact bytecode. this breaks applications that rely on not-yet-deployed "counterfactual contracts", erc-4337 wallets, eip-2470 singletons and many other applications. handwritten evm bytecode is more difficult to use. many applications use handwritten evm bytecode in some parts for efficiency. type 4 systems may not support it, though there are ways to implement limited evm bytecode support to satisfy these use cases without going through the effort of becoming a full-on type 3 zk-evm. lots of debugging infrastructure cannot be carried over, because such infrastructure runs over the evm bytecode. that said, this disadvantage is mitigated by the greater access to debugging infrastructure from "traditional" high-level or intermediate languages (eg. llvm). developers should be mindful of these issues. who's building it? zksync is a type 4 system, though it may add compatibility for evm bytecode over time. nethermind's warp project is building a compiler from solidity to starkware's cairo, which will turn starknet into a de-facto type 4 system. the future of zk-evm types the types are not unambiguously "better" or "worse" than other types. rather, they are different points on the tradeoff space: lower-numbered types are more compatible with existing infrastructure but slower, and higher-numbered types are less compatible with existing infrastructure but faster. in general, it's healthy for the space that all of these types are being explored. additionally, zk-evm projects can easily start at higher-numbered types and jump to lower-numbered types (or vice versa) over time. for example: a zk-evm could start as type 3, deciding not to include some features that are especially hard to zk-prove. later, they can add those features over time, and move to type 2. a zk-evm could start as type 2, and later become a hybrid type 2 / type 1 zk-evm, by providing the possibility of operating either in full ethereum compatibility mode or with a modified state tree that can be proven faster. scroll is considering moving in this direction. what starts off as a type 4 system could become type 3 over time by adding the ability to process evm code later on (though developers would still be encouraged to compile direct from high-level languages to reduce fees and prover times) a type 2 or type 3 zk-evm can become a type 1 zk-evm if ethereum itself adopts its modifications in an effort to become more zk-friendly. a type 1 or type 2 zk-evm can become a type 3 zk-evm by adding a precompile for verifying code in a very zk-snark-friendly language. this would give developers a choice between ethereum compatibility and speed. this would be type 3, because it breaks perfect evm equivalence, but for practical intents and purposes it would have a lot of the benefits of type 1 and 2. the main downside might be that some developer tooling would not understand the zk-evm's custom precompiles, though this could be fixed: developer tools could add universal precompile support by supporting a config format that includes an evm code equivalent implementation of the precompile. personally, my hope is that everything becomes type 1 over time, through a combination of improvements in zk-evms and improvements to ethereum itself to make it more zk-snark-friendly. in such a future, we would have multiple zk-evm implementations which could be used both for zk rollups and to verify the ethereum chain itself. theoretically, there is no need for ethereum to standardize on a single zk-evm implementation for l1 use; different clients could use different proofs, so we continue to benefit from code redundancy. however, it is going to take quite some time until we get to such a future. in the meantime, we are going to see a lot of innovation in the different paths to scaling ethereum and ethereum-based zk-rollups. dark mode toggle why proof of stake (nov 2020) 2020 nov 06 see all posts there are three key reasons why pos is a superior blockchain security mechanism compared to pow. pos offers more security for the same cost the easiest way to see this is to put proof of stake and proof of work side by side, and look at how much it costs to attack a network per $1 per day in block rewards. gpu-based proof of work you can rent gpus cheaply, so the cost of attacking the network is simply the cost of renting enough gpu power to outrun the existing miners. for every $1 of block rewards, the existing miners should be spending close to $1 in costs (if they're spending more, miners will drop out due to being unprofitable, if they're spending less, new miners can join in and take high profits). hence, attacking the network just requires temporarily spending more than $1 per day, and only for a few hours. total cost of attack: ~$0.26 (assuming 6-hour attack), potentially reduced to zero as the attacker receives block rewards asic-based proof of work asics are a capital cost: you buy an asic once and you can expect it to be useful for ~2 years before it wears out and/or is obsoleted by newer and better hardware. if a chain gets 51% attacked, the community will likely respond by changing the pow algorithm and your asic will lose its value. on average, mining is ~1/3 ongoing costs and ~2/3 capital costs (see here for some sources). hence, per $1 per day in reward, miners will be spending ~$0.33 per day on electricity+maintenance and ~$0.67 per day on their asic. assuming an asic lasts ~2 years, that's $486.67 that a miner would need to spend on that quantity of asic hardware. total cost of attack: $486.67 (asics) + $0.08 (electricity+maintenance) = $486.75 that said, it's worth noting that asics provide this heightened level of security against attacks at a high cost of centralization, as the barriers to entry to joining become very high. proof of stake proof of stake is almost entirely capital costs (the coins being deposited); the only operating costs are the cost of running a node. now, how much capital are people willing to lock up to get $1 per day of rewards? unlike asics, deposited coins do not depreciate, and when you're done staking you get your coins back after a short delay. hence, participants should be willing to pay much higher capital costs for the same quantity of rewards. let's assume that a ~15% rate of return is enough to motivate people to stake (that is the expected eth2 rate of return). then, $1 per day of rewards will attract 6.667 years' worth of returns in deposits, or $2433. hardware and electricity costs of a node are small; a thousand-dollar computer can stake for hundreds of thousands of dollars in deposits, and ~$100 per months in electricity and internet is sufficient for such an amount. but conservatively, we can say these ongoing costs are ~10% of the total cost of staking, so we only have $0.90 per day of rewards that end up corresponding to capital costs, so we do need to cut the above figure by ~10%. total cost of attack: $0.90/day * 6.667 years = $2189 in the long run, this cost is expected to go even higher, as staking becomes more efficient and people become comfortable with lower rates of return. i personally expect this number to eventually rise to something like $10000. note that the only "cost" being incurred to get this high level of security is just the inconvenience of not being able to move your coins around at will while you are staking. it may even be the case that the public knowledge that all these coins are locked up causes the value of the coin to rise, so the total amount of money floating around in the community, ready to make productive investments etc, remains the same! whereas in pow, the "cost" of maintaining consensus is real electricity being burned in insanely large quantities. higher security or lower costs? note that there are two ways to use this 5-20x gain in security-per-cost. one is to keep block rewards the same but benefit from increased security. the other is to massively reduce block rewards (and hence the "waste" of the consensus mechanism) and keep the security level the same. either way is okay. i personally prefer the latter, because as we will see below, in proof of stake even a successful attack is much less harmful and much easier to recover from than an attack on proof of work! attacks are much easier to recover from in proof of stake in a proof of work system, if your chain gets 51% attacked, what do you even do? so far, the only response in practice has been "wait it out until the attacker gets bored". but this misses the possibility of a much more dangerous kind of attack called a spawn camping attack, where the attacker attacks the chain over and over again with the explicit goal of rendering it useless. in a gpu-based system, there is no defense, and a persistent attacker may quite easily render a chain permanently useless (or more realistically, switches to proof of stake or proof of authority). in fact, after the first few days, the attacker's costs may become very low, as honest miners will drop out since they have no way to get rewards while the attack is going on. in an asic-based system, the community can respond to the first attack, but continuing the attack from there once again becomes trivial. the community would meet the first attack by hard-forking to change the pow algorithm, thereby "bricking" all asics (the attacker's and honest miners'!). but if the attacker is willing to suffer that initial expense, after that point the situation reverts to the gpu case (as there is not enough time to build and distribute asics for the new algorithm), and so from there the attacker can cheaply continue the spawn camp inevitably. in the pos case, however, things are much brighter. for certain kinds of 51% attacks (particularly, reverting finalized blocks), there is a built-in "slashing" mechanism in the proof of stake consensus by which a large portion of the attacker's stake (and no one else's stake) can get automatically destroyed. for other, harder-to-detect attacks (notably, a 51% coalition censoring everyone else), the community can coordinate on a minority user-activated soft fork (uasf) in which the attacker's funds are once again largely destroyed (in ethereum, this is done via the "inactivity leak mechanism"). no explicit "hard fork to delete coins" is required; with the exception of the requirement to coordinate on the uasf to select a minority block, everything else is automated and simply following the execution of the protocol rules. hence, attacking the chain the first time will cost the attacker many millions of dollars, and the community will be back on their feet within days. attacking the chain the second time will still cost the attacker many millions of dollars, as they would need to buy new coins to replace their old coins that were burned. and the third time will... cost even more millions of dollars. the game is very asymmetric, and not in the attacker's favor. proof of stake is more decentralized than asics gpu-based proof of work is reasonably decentralized; it is not too hard to get a gpu. but gpu-based mining largely fails on the "security against attacks" criterion that we mentioned above. asic-based mining, on the other hand, requires millions of dollars of capital to get into (and if you buy an asic from someone else, most of the time, the manufacturing company gets the far better end of the deal). this is also the correct answer to the common "proof of stake means the rich get richer" argument: asic mining also means the rich get richer, and that game is even more tilted in favor of the rich. at least in pos the minimum needed to stake is quite low and within reach of many regular people. additionally, proof of stake is more censorship resistant. gpu mining and asic mining are both very easy to detect: they require huge amounts of electricity consumption, expensive hardware purchases and large warehouses. pos staking, on the other hand, can be done on an unassuming laptop and even over a vpn. possible advantages of proof of work there are two primary genuine advantages of pow that i see, though i see these advantages as being fairly limited. proof of stake is more like a "closed system", leading to higher wealth concentration over the long term in proof of stake, if you have some coin you can stake that coin and get more of that coin. in proof of work, you can always earn more coins, but you need some outside resource to do so. hence, one could argue that over the long term, proof of stake coin distributions risk becoming more and more concentrated. the main response to this that i see is simply that in pos, the rewards in general (and hence validator revenues) will be quite low; in eth2, we are expecting annual validator rewards to equal ~0.5-2% of the total eth supply. and the more validators are staking, the lower interest rates get. hence, it would likely take over a century for the level of concentration to double, and on such time scales other pressures (people wanting to spend their money, distributing their money to charity or among their children, etc.) are likely to dominate. proof of stake requires "weak subjectivity", proof of work does not see here for the original intro to the concept of "weak subjectivity". essentially, the first time a node comes online, and any subsequent time a node comes online after being offline for a very long duration (ie. multiple months), that node must find some third-party source to determine the correct head of the chain. this could be their friend, it could be exchanges and block explorer sites, the client developers themselves, or many other actors. pow does not have this requirement. however, arguably this is a very weak requirement; in fact, users need to trust client developers and/or "the community" to about this extent already. at the very least, users need to trust someone (usually client developers) to tell them what the protocol is and what any updates to the protocol have been. this is unavoidable in any software application. hence, the marginal additional trust requirement that pos imposes is still quite low. but even if these risks do turn out to be significant, they seem to me to be much lower than the immense gains that pos sytems get from their far greater efficiency and their better ability to handle and recover from attacks. see also: my previous pieces on proof of stake. proof of stake faq a proof of stake design philosophy blockchain scalability: chain-fibers redux | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search blockchain scalability: chain-fibers redux posted by gavin wood on april 5, 2015 research & development history i came up with the first seed of this idea while chatting to janislav malahov in berlin in spring 2014. unfortunately, the original article i wrote was lost along with my laptop when it was stolen in vienna. after chatting over the principles with vitalik more recently, we made a number of alterations and formalisations, mainly to the validation and the sub-state cutting mechanisms. what follows is a fairly complete illustration of one particular possible plan for block chain scalability in a later version of ethereum. since this is by no means a final proposal, there is a github wiki page that will track the progress on this particular idea. overview the basic idea of chain-fibers is unchanged from a year ago; split the state-space up into strata and have separate transaction collators specialising in one or a number of state sub-spaces. transactions requiring interactions from many a subspace would be accordingly more expensive (since collators would have to maintain presence on multiple chains) and take longer to execute (since there is a lesser chance that any given block would contain a superset of the transaction's subspaces). validity of a transaction is verifiable in isolation through the provision of comprehensive merkle proofs to its inputs alongside it in the block in which it is included. the subtleties lie in precisely what governs the division of subspaces (my original proposal included the automated splitting, merging and rotation of subspace-divisions in order to best deliver internal coherency), how security is maintained within comparatively worthless subspaces and how this can play well with proof-of-stake (the original was based upon a master pow chain, feeding off an idea put forward by max kaye in early 2014 to disassociate block chain archival from transition semantics). basic idea is to have a number of chains (e.g. n), each detailing the state-transitions for only a strata of the entire system state (i.e. a state subspace). following from programming terminology, these might be termed "fibers". accounts thus belong to a subspace and as such a single fiber; the fiber to which they belong can be determined simply from the first log2(n) bits of the address. n can increase or decrease, and is a value maintained within the housekeeping information on the "master chain". the master chain in maintained by a set of bonded validators v, with the number of validators proportional to n. a random selection of validators validate each block produced, and validators ultimately vote to form consensus over the master chain. each block of the master chain maintains a reference to the header of each fiber. transaction collators produce blocks (accepting fees from transactors), and pay validators some of the fees collected to include the hash of their block in the main chain. blocks are produced across a particular "home set" of fibers; this is basically just the set of fibers of which they maintain the state trie. their blocks may involve transactions over one or many of these fibers, though none outside their "home set". "fishermen" is a term given to freelance checkers. since block validation and availability are both important, and since it is possible that sets of validators may be contractually bribed, it is important to have a mechanism to involve additional rational individuals in acting as "whistle-blowers" to avoid bogging the other validators needlessly checking all blocks. the fishermen basically pay to attempt to convince a quorum of validators that a previously validated block is invalid (or unavailable, which we assume is equivalent). if a fisherman demonstrates a validator (or, more likely, set of validators) acted in a dishonourable fashion, then they get to claim all of their bonds. to avoid dosing the validators with spurious challenges, a fee is payable. schematic sorry for the not-quite ascii-art. i'm not quite as 1337 at inkscape as vitalik. transactors ==tx+fee==> collators ==block+fee==> validators make transaction validate transaction, random selection chosen to audit produce comprehensive merkle tx/psr/cmp contents & availability, proof and post state root, all placed in pos-consensus master block collate into x-fiber block fishermen ==challenge+fee==> validators search for invalid or a selection adjudicate challenge unavailable x-fiber blocks transactors transactors are pretty much exactly the same as in ethereum 1.0 they are the users of the system. transactors: make transaction transactors make a transaction much like they do in the existing ethereum system. one or two minor differences addresses can be used as a distance metric; those sharing the same number of initial bits are considered "closer", which means a greater certainty into the future that they will continue to be contained in the same state subspace. contracts are naturally created in the same state subspace as the creator. transactions, like collators, operate over a number of fibers; perhaps one perhaps all, probably somewhere in between. submission to collators may be directed through fiber sub-network overlays. submission and payment to the collators happens much as existing transaction submission to miners happens in ethereum 1.0. collators collators maintain presence on at least two peer sub-network overlays; the validators overlay, and one or more fiber overlays. the fiber overlays may provide directed transaction propogation. collators "collate" on a set of fibers. they maintain a full fiber-chain for each fiber they collate over, and can accept all transactions that involve any combination of their fiber set. the greater this combination, then the greater their "transaction net", but the greater their overall disk/memory footprint. collators: validate transaction on receipt of a transaction, they go through the usual ethereum 1.0 rites of checking payment is enough, initial balances &c. once basic validation is done, they attempt to execute it, throwing it out if it touches any fiber that is not part of collator's fiber set. collators: produce comprehensive merkle proof and post state root collators provide each post-state-root (as is found in the transaction receipt of ethereum 1.0) and append to the block merkle proofs and associated hints (e.g. contract code) for all inputs (balance, nonce, state, code) from all subspaces that are required for the evaluation of each transaction from a previously known post-state-root. this allows an auditor to, without anything other than the previous post-state-root for each fiber, determine the validity of the block. collators: collate into x-fiber block a cross fiber block is created from the total information collated. this includes transactions, transaction receipts (post-state-roots), comprehensive merkle-proofs and associated hash-hints. this block does not include any consensus-specific information such as timestamping, uncles &c. validators validators (who might be better named auditors) are bonded particpants, chosen regularly from the highest bidders, who take a small fee for the ultimate maintenence of the network. their job, as a whole, is to form a judiciary and ultimate authority over the validity and transaction contents of the chain. we generally assume that they are mostly benevolent and cannot all be bribed. being bonded, validators may also be called to audit and stake their bond on an opinion over validity or information-availability. validators: all placed in pos-consensus master block they maintain signing control over the master chain. the master chain (mc) encodes all pos/consensus stuff like timestamping and includes its own little state root for recording validator's bond balances, ongoing challenges, fiber block header-hashes and any other housekeeping information. each master block (mb), a set of collated x-fiber blocks (xbs) are taken; these must be non-overlapping, so that each fiber belongs to only a single xb. validators: random selection chosen to audit tx/psr/cmp contents & availability for each mb we have a number of xsbs referenced from the mb's trie. each fiber is assigned a randomly selected set of validators, and the validators must review whatever xb contains their assigned fiber. validation includes attaining the xb, finding the previous psrs for each of the fibers (placed in the mb) and checking that the proofs in its cmp, cover all required inputs to the transactions collated within and that the psr is indeed the final state root when all are executed. the block is considered valid iff all assigned validators sign it. signing it is considered an assertion that the block contents are both valid and available for a probabilistically long "challenge period" in which a fisherman may challenge. any challenge to the block's validity which is ultimately upheld by a full consensus of a randomly selected set of validators (ultimately ending with a majority vote, should it be doggedly contested) will mean the instant loss of the bond. fishermen fishermen (who might be called bounty hunters) are the freelance error-checkers of the system. the watch the validators in the hope that they can find wrong-doing. to help guarantee presence, payouts are designed to be huge. the costs of challenging are small but not insignificant. fishermen: search for invalid or unavailable x-fiber blocks they check the x-fiber blocks looking for validity errors and/or inavailability of data. when they find an invalid block or unavailable data, they launch a challenge (for a small fee, paid to validators) in the hope that a sufficiently large portion of validators will concur. if they succeed and validators ultimately uphold the challenge, then they receive the bonds of all validators who had previously asserted validity/availability of the information. fishermen's challenge fisherman finds an invalid/unavailable block not yet outside its "challenge period" (10-30 blocks); pays a fee, submits a challenge transaction into the master chain; a randomly selected set of validators (e.g. of order e.g. sqrt(n)) ++ any validators that self-select (through doubling their bond), check the block that was challenged; each votes y or n to the block's validity; if n, the validator receives a small payment pn. if y, the validator stakes their bond, though receives a larger payment py (perhaps py = 2pn). the outcome of the challenge (probably accumulated into the following block) is: if more than 66% of validators vote y (valid), then the challenge ends. the fisherman loses their fee, but may reinitiate a challenge. if at least one validator votes y (valid), then the challenge continues with a second, larger set of randomly selected validators. all bonds are staked. if all validators vote n (invalid), then the block is recorded as invalid and the fishermen receives the bond of all validators that have asserted the blocks validity. this is a very large payoff. note: if the set includes all validators, then it's a simple majority-carries rule. other differences all addresses are contained in a lookup table unique to each state subspace; this means they can be referenced through a small number of bits and avoid large amounts of wasted entropy in the rlp for proofs &c. notes once a block is out of the challenge period, it is considered unassailable. if it does turn out to be bad, then it must be fixed in the same way as a protocol upgrade. as such it is likely that validators and other large stakeholder would act as fishermen to protect their investment. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle the roads not taken 2022 mar 29 see all posts the ethereum protocol development community has made a lot of decisions in the early stages of ethereum that have had a large impact on the project's trajectory. in some cases, ethereum developers made conscious decisions to improve in some place where we thought that bitcoin erred. in other places, we were creating something new entirely, and we simply had to come up with something to fill in a blank but there were many somethings to choose from. and in still other places, we had a tradeoff between something more complex and something simpler. sometimes, we chose the simpler thing, but sometimes, we chose the more complex thing too. this post will look at some of these forks-in-the-road as i remember them. many of these features were seriously discussed within core development circles; others were barely considered at all but perhaps really should have been. but even still, it's worth looking at what a different ethereum might have looked like, and what we can learn from this going forward. should we have gone with a much simpler version of proof of stake? the gasper proof of stake that ethereum is very soon going to merge to is a complex system, but a very powerful system. some of its properties include: very strong single-block confirmations as soon as a transaction gets included in a block, usually within a few seconds that block gets solidified to the point that it cannot be reverted unless either a large fraction of nodes are dishonest or there is extreme network latency. economic finality once a block gets finalized, it cannot be reverted without the attacker having to lose millions of eth to being slashed. very predictable rewards validators reliably earn rewards every epoch (6.4 minutes), reducing incentives to pool support for very high validator count unlike most other chains with the above features, the ethereum beacon chain supports hundreds of thousands of validators (eg. tendermint offers even faster finality than ethereum, but it only supports a few hundred validators) but making a system that has these properties is hard. it took years of research, years of failed experiments, and generally took a huge amount of effort. and the final output was pretty complex. if our researchers did not have to worry so much about consensus and had more brain cycles to spare, then maybe, just maybe, rollups could have been invented in 2016. this brings us to a question: should we really have had such high standards for our proof of stake, when even a much simpler and weaker version of proof of stake would have been a large improvement over the proof of work status quo? many have the misconception that proof of stake is inherently complex, but in reality there are plenty of proof of stake algorithms that are almost as simple as nakamoto pow. nxt proof of stake existed since 2013 and would have been a natural candidate; it had issues but those issues could easily have been patched, and we could have had a reasonably well-working proof of stake from 2017, or even from the beginning. the reason why gasper is more complex than these algorithms is simply that it tries to accomplish much more than they do. but if we had been more modest at the beginning, we could have focused on achieving a more limited set of objectives first. proof of stake from the beginning would in my opinion have been a mistake; pow was helpful in expanding the initial issuance distribution and making ethereum accessible, as well as encouraging a hobbyist community. but switching to a simpler proof of stake in 2017, or even 2020, could have led to much less environmental damage (and anti-crypto mentality as a result of environmental damage) and a lot more research talent being free to think about scaling. would we have had to spend a lot of resources on making a better proof of stake eventually? yes. but it's increasingly looking like we'll end up doing that anyway. the de-complexification of sharding ethereum sharding has been on a very consistent trajectory of becoming less and less complex since the ideas started being worked on in 2014. first, we had complex sharding with built-in execution and cross-shard transactions. then, we simplified the protocol by moving more responsibilities to the user (eg. in a cross-shard transaction, the user would have to separately pay for gas on both shards). then, we switched to the rollup-centric roadmap where, from the protocol's point of view, shards are just blobs of data. finally, with danksharding, the shard fee markets are merged into one, and the final design just looks like a non-sharded chain but where some data availability sampling magic happens behind the scenes to make sharded verification happen. sharding in 2015 sharding in 2022 but what if we had gone the opposite path? well, there actually are ethereum researchers who heavily explored a much more sophisticated sharding system: shards would be chains, there would be fork choice rules where child chains depend on parent chains, cross-shard messages would get routed by the protocol, validators would be rotated between shards, and even applications would get automatically load-balanced between shards! the problem with that approach: those forms of sharding are largely just ideas and mathematical models, whereas danksharding is a complete and almost-ready-for-implementation spec. hence, given ethereum's circumstances and constraints, the simplification and de-ambitionization of sharding was, in my opinion, absolutely the right move. that said, the more ambitious research also has a very important role to play: it identifies promising research directions, even the very complex ideas often have "reasonably simple" versions of those ideas that still provide a lot of benefits, and there's a good chance that it will significantly influence ethereum's protocol development (or even layer-2 protocols) over the years to come. more or less features in the evm? realistically, the specification of the evm was basically, with the exception of security auditing, viable for launch by mid-2014. however, over the next few months we continued actively exploring new features that we felt might be really important for a decentralized application blockchain. some did not go in, others did. we considered adding a post opcode, but decided against it. the post opcode would have made an asynchronous call, that would get executed after the rest of the transaction finishes. we considered adding an alarm opcode, but decided against it. alarm would have functioned like post, except executing the asynchronous call in some future block, allowing contracts to schedule operations. we added logs, which allow contracts to output records that do not touch the state, but could be interpreted by dapp interfaces and wallets. notably, we also considered making eth transfers emit a log, but decided against it the rationale being that "people will soon switch to smart contract wallets anyway". we considered expanding sstore to support byte arrays, but decided against it, because of concerns about complexity and safety. we added precompiles, contracts which execute specialized cryptographic operations with native implementations at a much cheaper gas cost than can be done in the evm. in the months right after launch, state rent was considered again and again, but was never included. it was just too complicated. today, there are much better state expiry schemes being actively explored, though stateless verification and proposer/builder separation mean that it is now a much lower priority. looking at this today, most of the decisions to not add more features have proven to be very good decisions. there was no obvious reason to add a post opcode. an alarm opcode is actually very difficult to implement safely: what happens if everyone in blocks 1...99999 sets an alarm to execute a lot of code at block 100000? will that block take hours to process? will some scheduled operations get pushed back to later blocks? but if that happens, then what guarantees is alarm even preserving? sstore for byte arrays is difficult to do safely, and would have greatly expanded worst-case witness sizes. the state rent issue is more challenging: had we actually implemented some kind of state rent from day 1, we would not have had a smart contract ecosystem evolve around a normalized assumption of persistent state. ethereum would have been harder to build for, but it could have been more scalable and sustainable. at the same time, the state expiry schemes we had back then really were much worse than what we have now. sometimes, good ideas just take years to arrive at and there is no better way around that. alternative paths for log log could have been done differently in two different ways: we could have made eth transfers auto-issue a log. this would have saved a lot of effort and software bug issues for exchanges and many other users, and would have accelerated everyone relying on logs that would have ironically helped smart contract wallet adoption. we could have not bothered with a log opcode at all, and instead made it an erc: there would be a standard contract that has a function submitlog and uses the technique from the ethereum deposit contract to compute a merkle root of all logs in that block. either eip-2929 or block-scoped storage (equivalent to tstore but cleared after the block) would have made this cheap. we strongly considered (1), but rejected it. the main reason was simplicity: it's easier for logs to just come from the log opcode. we also (very wrongly!) expected most users to quickly migrate to smart contract wallets, which could have logged transfers explicitly using the opcode. was not considered, but in retrospect it was always an option. the main downside of (2) would have been the lack of a bloom filter mechanism for quickly scanning for logs. but as it turns out, the bloom filter mechanism is too slow to be user-friendly for dapps anyway, and so these days more and more people use thegraph for querying anyway. on the whole, it seems very possible that either one of these approaches would have been superior to the status quo. keeping log outside the protocol would have kept things simpler, but if it was inside the protocol auto-logging all eth transfers would have made it more useful. today, i would probably favor the eventual abolition of the log opcode from the evm. what if the evm was something totally different? there were two natural very different paths that the evm could have taken: make the evm be a higher-level language, with built-in constructs for variables, if-statements, loops, etc. make the evm be a copy of some existing vm (llvm, wasm, etc) the first path was never really considered. the attraction of this path is that it could have made compilers simpler, and allowed more developers to code in evm directly. it could have also made zk-evm constructions simpler. the weakness of the path is that it would have made evm code structurally more complicated: instead of being a simple list of opcodes in a row, it would have been a more complicated data structure that would have had to be stored somehow. that said, there was a missed opportunity for a best-of-both-worlds: some evm changes could have given us a lot of those benefits while keeping the basic evm structure roughly as is: ban dynamic jumps and add some opcodes designed to support subroutines (see also: eip-2315), allow memory access only on 32-byte word boundaries, etc. the second path was suggested many times, and rejected many times. the usual argument for it is that it would allow programs to compile from existing languages (c, rust, etc) into the evm. the argument against has always been that given ethereum's unique constraints it would not actually provide any benefits: existing compilers from high-level languages tend to not care about total code size, whereas blockchain code must optimize heavily to cut down every byte of code size we need multiple implementations of the vm with a hard requirement that two implementations never process the same code differently. security-auditing and verifying this on code that we did not write would be much harder. if the vm specification changes, ethereum would have to either always update along with it or fall more and more out-of-sync. hence, there probably was never a viable path for the evm that's radically different from what we have today, though there are lots of smaller details (jumps, 64 vs 256 bit, etc) that could have led to much better outcomes if they were done differently. should the eth supply have been distributed differently? the current eth supply is approximately represented by this chart from etherscan: about half of the eth that exists today was sold in an open public ether sale, where anyone could send btc to a standardized bitcoin address, and the initial eth supply distribution was computed by an open-source script that scans the bitcoin blockchain for transactions going to that address. most of the remainder was mined. the slice at the bottom, the 12m eth marked "other", was the "premine" a piece distributed between the ethereum foundation and ~100 early contributors to the ethereum protocol. there are two main criticisms of this process: the premine, as well as the fact that the ethereum foundation received the sale funds, is not credibly neutral. a few recipient addresses were hand-picked through a closed process, and the ethereum foundation had to be trusted to not take out loans to recycle funds received furing the sale back into the sale to give itself more eth (we did not, and no one seriously claims that we have, but even the requirement to be trusted at all offends some). the premine over-rewarded very early contributors, and left too little for later contributors. 75% of the premine went to rewarding contributors for their work before launch, and post-launch the ethereum foundation only had 3 million eth left. within 6 months, the need to sell to financially survive decreased that to around 1 million eth. in a way, the problems were related: the desire to minimize perceptions of centralization contributed to a smaller premine, and a smaller premine was exhausted more quickly. this is not the only way that things could have been done. zcash has a different approach: a constant 20% of the block reward goes to a set of recipients hard-coded in the protocol, and the set of recipients gets re-negotiated every 4 years (so far this has happened once). this would have been much more sustainable, but it would have been much more heavily criticized as centralized (the zcash community seems to be more openly okay with more technocratic leadership than the ethereum community). one possible alternative path would be something similar to the "dao from day 1" route popular among some defi projects today. here is a possible strawman proposal: we agree that for 2 years, a block reward of 2 eth per block goes into a dev fund. anyone who purchases eth in the ether sale could specify a vote for their preferred distribution of the dev fund (eg. "1 eth per block to the ethereum foundation, 0.4 eth to the consensys research team, 0.2 eth to vlad zamfir...") recipients that got voted for get a share from the dev fund equal to the median of everyone's votes, scaled so that the total equals 2 eth per block (median is to prevent self-dealing: if you vote for yourself you get nothing unless you get at least half of other purchasers to mention you) the sale could be run by a legal entity that promises to distribute the bitcoin received during the sale along the same ratios as the eth dev fund (or burned, if we really wanted to make bitcoiners happy). this probably would have led to the ethereum foundation getting a lot of funding, non-ef groups also getting a lot of funding (leading to more ecosystem decentralization), all without breaking credible neutrality one single bit. the main downside is of course that coin voting really sucks, but pragmatically we could have realized that 2014 was still an early and idealistic time and the most serious downsides of coin voting would only start coming into play long after the sale ends. would this have been a better idea and set a better precedent? maybe! though realistically even if the dev fund had been fully credibly neutral, the people who yell about ethereum's premine today may well have just started yelling twice as hard about the dao fork instead. what can we learn from all this? in general, it sometimes feels to me like ethereum's biggest challenges come from balancing between two visions a pure and simple blockchain that values safety and simplicity, and a highly performant and functional platform for building advanced applications. many of the examples above are just aspects of this: do we have fewer features and be more bitcoin-like, or more features and be more developer-friendly? do we worry a lot about making development funding credibly neutral and be more bitcoin-like, or do we just worry first and foremost about making sure devs are rewarded enough to make ethereum great? my personal dream is to try to achieve both visions at the same time a base layer where the specification becomes smaller each year than the year before it, and a powerful developer-friendly advanced application ecosystem centered around layer-2 protocols. that said, getting to such an ideal world takes a long time, and a more explicit realization that it would take time and we need to think about the roadmap step-by-step would have probably helped us a lot. today, there are a lot of things we cannot change, but there are many things that we still can, and there is still a path solidly open to improving both functionality and simplicity. sometimes the path is a winding one: we need to add some more complexity first to enable sharding, which in turn enables lots of layer-2 scalability on top. that said, reducing complexity is possible, and ethereum's history has already demonstrated this: eip-150 made the call stack depth limit no longer relevant, reducing security worries for contract developers. eip-161 made the concept of an "empty account" as something separate from an account whose fields are zero no longer exist. eip-3529 removed part of the refund mechanism and made gas tokens no longer viable. ideas in the pipeline, like verkle trees, reduce complexity even further. but the question of how to balance the two visions better in the future is one that we should start more actively thinking about. dark mode toggle 7ème tour des subventions gitcoin rétrospective 2020 oct 18 see all posts le septième tour des subventions gitcoin s'est terminé avec succès! ce tour a connu une augmentation de l'intérêt pour gitcoin sans précédent, ce qui s'est traduit par 274.830 $ de contributions et 450.000 $ en abondement répartis sur 857 projets. la structure des catégories a été une fois de plus modifiée; cette fois-ci, divisée entre "tech dapp", "infrastructure" et "communauté". voici les résultats : defi s'associe au mouvement ! lors de ce tour, le nombre de projets associés s'est avéré plus élevé qu'auparavant. en plus des projets habituels par la fondation ethereum ainsi que par quelques autres acteurs, se sont ajoutés pour la première fois différents projets defi : les participants étaient : * chainlink, un projet oracle smart contract * optimisme, un rollup optimiste de la couche 2 * la fondation ethereum * balancer, une bourse décentralisée * synthetix, une plateforme d'actifs synthétiques * yann, une plateforme de prêt garantis * three arrows capital, un fonds d'investissement * defiance capital, un autre fonds d'investissement * future fund, qui n'est absolument pas un fonds d'investissement ! (/s) * $meme, un memecoin * yam, un projet defi * quelques contributeurs individuels : ferretpatrol, bantg, mariano conti, robert leshner, eric conner, 10b576da0 ensemble, les différents acteurs ont contribué de manière significative au financement par abondement dont une portion a été utilisé lors de ce round et une autre alloué aux « fonds d'urgence » pour les prochains tours, au cas où les futurs participants seraient moins enclin à contribuer. il s'agit d'une étape importante pour l'écosystème, cela montre que les subventions gitcoin s'élargissent, indépendamment d'un petit cercle d'investisseurs, dans la direction de quelque chose de plus durable. on peut maintenant se demander ce qui pousse ces projets à contribuer, et surtout est-ce que cela va durer ? voici un certain nombre de motivations possibles agissant probablement à des degrés divers : les gens sont, dans une certaine mesure, naturellement altruistes, et durant ce tour, les acteurs de projets defi devenu inopinément riches pour la première fois en raison d'une hausse rapide de la valeur des tokens et des intérêts, ont ainsi donné une partie de cette aubaine car cela semblait être tout naturellement une « bonne chose à faire » par défaut, beaucoup dans la communauté critiquent les projets defi, les considérant comme des casinos improductifs créant une image négative des valeurs qu'ethereum représente. contribuer aux biens publics est un moyen direct pour un projet defi de montrer qu'il peut avoir un impact positif sur l'écosystème et le rendre meilleur négativisme mis à part, la defi est un marché concurrentiel qui dépend fortement du soutien communautaire et des effets de réseau, et il est donc crucial pour un projet de se faire des amis dans l'écosystème les plus grands projets defi tirent leurs bénéfices des biens publics et il est dans leur propre intérêt d'y contribuer en retour il y a un degré élevé de propriété commune entre les projets defi (les détenteurs d'un jeton détiennent également d'autres jetons et détiennent eth), et donc même si ce n'est pas strictement dans l'intérêt d'un projet de faire un large don, les détenteurs de jetons de ce projet poussent le projet à contribuer parce que le bénéfice est mutuel pour les détenteurs, le projet mais aussi les autres projets dont ils détiennent des jetons. la question qui reste à poser est la suivante : dans quelle mesure ces incitations seront-elles durables? les incitations altruistes et les relations publiques sont-elles importantes au point de justifier une explosion de dons seulement ponctuelle ou pourraient-elles devenir plus durables? pourrions-nous nous attendre à ce que, par exemple, 2-3 millions de dollars par an puissent être dépensés en fonds de contrepartie quadratique à partir d'aujourd'hui? si tel est le cas, ce serait une excellente nouvelle pour la diversification et la démocratisation du financement des biens publics dans l'écosystème ethereum. où se cachent les fauteurs de troubles? un résultat curieux de ce round et du précédent est la perte d'importance des bénéficiaires de subventions communautaires "controversées" lors des tours précédents. en théorie, nous aurions dû les voir continuer à obtenir le soutien de leurs partisans, leurs détracteurs ne pouvant rien y faire. en pratique il en a été autrement. même le zero knowledge podcast, un excellent podcast mais destiné à un public relativement plus petit et plus technique, a reçu une importante contribution. que s'est-il passé? comment la distribution auprès des médias s'est-elle améliorée en termes de qualité de façon autonome? le mécanisme s'auto-corrige-t-il mieux que nous ne le pensions? surplus de financement ce tour est le premier où les principaux récipiendaires de tout bord ont reçu une somme conséquente. du côté de l'infrastructure, le projet white hat hacking (essentiellement un fonds de don à samczsun) a reçu un total de 39.258 $ et le bankless podcast a obtenu 47.620 $. nous pourrions nous demander : est-ce que les principaux bénéficiaires reçoivent trop de fonds? pour être clair, je pense qu'il est assez inapproprié d'essayer de créer une norme morale selon laquelle le salaires des contributeurs de biens publics devraient être plafonné jusqu'à un certain niveau sans pouvoir gagner beaucoup plus. les gens qui lancent des monnaies génèrent d'énormes gains; il est donc tout à fait naturel et juste pour les contributeurs de biens publics d'obtenir également cette possibilité (et en outre, les chiffres de ce tour se traduisent par environ ~200.000$ par an, ce qui en soit n'est pas si élevé). cependant, on peut se poser une question plus limitée et plus pragmatique : compte tenu de la structure actuelle des commissions, est-ce qu'un montant supplémentaire de 1$ entre les mains d'un des principaux contributeurs a moins de valeur qu'un 1$ entre les mains d'un autre projet encore sous-financé mais tout aussi précieux? turbogeth, nethermind, radicalxchange et beaucoup d'autres projets pourraient encore faire beaucoup avec un dollar de plus. pour la première fois, les montants correspondants sont suffisamment élevés pour que le problème se pose. si les montants correspondants augmentent encore davantage, l'écosystème va-t-il être en mesure d'allouer correctement les fonds et d'éviter le surfinancement de certains projets? dans le cas contraire, cela serait-il si mauvais? peut-être que la possibilité de devenir le centre d'attention pour un tour et de gagner 500.000 $ fera partie de l'incitation qui motivera les contributeurs indépendants à participer aux biens publics ! difficile à dire, mais la réalisation de cette expérience à grande échelle nous le révélera. parlons des catégories... le concept de catégories tel qu'il est actuellement mis en œuvre dans les subventions gitcoin est un peu étrange. chaque catégorie a un montant d'abondement total fixe divisé entre les projets en faisant partie. fondamentalement, ce que ce mécanisme signifie c'est qu'en aval nous pouvons faire confiance à la communauté pour choisir entre les projets d'une catégorie, mais qu'en amont nous avons besoin d'un jugement technocratique distinct pour juger de la répartition des fonds entre les différentes catégories. mais le paradoxe commence ici. au cours du septième tour, une fonction "collections" a été introduite à mi-chemin : si vous cliquez sur « ajouter au panier » sur une collection, vous ajoutez immédiatement tout ce qui se trouve dans la collection à votre panier. c'est étrange parce que ce mécanisme semble envoyer exactement le message contraire: les utilisateurs qui ne comprennent pas bien la nuance peuvent choisir d'allouer des fonds à des catégories entières, mais (à moins qu'ils ne modifient manuellement les montants) ils ne devraient pas prendre de décisions spécifiques à chaque catégorie. qu'est-ce que cela signifie? suivons-nous le fantasme de la démocratie quadratique radicale pour la répartition entre les catégories, mais pas au sein de chacune d'entre elles? ma recommandation pour le 8ème tour est de réfléchir plus intensément aux défis philosophiques et d'adopter une approche fondée sur ces principes. une option serait d'avoir une cagnotte globale avec toutes les catégories et une couche d'interface pour les utilisateurs volontaires. une autre serait d'expérimenter avec encore plus "d'actions positives" pour supporter certaines catégories en particulier: par exemple, nous pourrions diviser la catégorie "communauté" en un pool correspondant à 25.000 $ pour chaque grande région mondiale (par exemple, amérique du nord + océanie, amérique latine, europe, afrique, moyen-orient, inde, est + asie du sud-est) pour essayer de donner un coup de pouce aux projets dans des zones plus négligées. il y a beaucoup de possibilités ici! l'idée d'un flux hybride serait que les cagnottes « ciblées » pourraient elles-mêmes être financées de façon quadratique lors du tour précédent! vérification de l'identité comme la collusion, les faux comptes et d'autres attaques sur les subventions gitcoin ont récemment augmenté, le 7ème tour a ajouté une option de vérification supplémentaire basée sur le graph-social décentralisé de brightid, qui à lui seul a amélioré la base utilisateurs du projet par un facteur de dix : c'est une bonne nouvelle car en plus d'aider la croissance de brightid, il soumet également le projet à l'épreuve du feu: il y a maintenant un intérêt à essayer de créer un grand nombre de faux comptes dessus! brightid va faire face à un défi difficile, celui de permettre aux utilisateurs réguliers de se joindre de façon raisonnablement facile tout en résistant aux attaques de faux comptes et doublons. j'ai hâte de les voir essayer de relever le défi! zk rollups pour plus d'évolutivité enfin, le 7ème tour a vu pour la première fois les subventions gitcoin expérimenter l'utilisation du zksync zk rollup pour réduire les frais de paiement: ce qui faut retenir ici est simplement que le rollup zk a en effet réussi à réduire les frais! l'expérience utilisateur a bien fonctionné. de nombreux projets utilisants des rollups optimistes et zk cherchent maintenant à collaborer avec des portefeuilles sur des intégrations directes, ce qui devrait rendre cette technique plus facile à utiliser et ce de façon plus sécurisée. conclusion le 7ème tour a été un tour pivot pour les subventions gitcoin. le financement de contrepartie est devenu beaucoup plus durable. les niveaux de financement sont maintenant suffisants pour financer avec succès les freelancers de façon quadratique, au point même de se demander si nous devons nous inquiéter d'un "surplus de financement"! la vérification de l'identité progresse. les paiements sont devenus beaucoup plus efficaces avec l'introduction du zksync zk rollup. j'ai hâte de voir les subventions se poursuivre pour de nombreux autres tours à l'avenir. ethereum builders: a proposed experiment | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum builders: a proposed experiment posted by mihai alisie on march 20, 2015 research & development hello, fellow ethereans. i am mihai alisie, one of the ethereum founders, and for the past year i’ve served as the vice president of the ethereum foundation and as director of ethereum switzerland. in the early days of the project i concentrated my attention on the community side and worked on the holons idea for a distributed physical infrastructure. for the past few months i've been contemplating what the next project step(s) should be and how we can create favorable conditions for a thriving community and ecosystem of smart contracts. in this post i will propose an experiment for the ethereum community – in a way, returning to the things i wanted to work on in the beginning of the project. the main idea behind this experiment is the making of a self organizing knowledge sharing network with potential ramifications in a variety of other areas. we’ll explore a few of the use cases towards the end of the article, but only in order to provide a starting point in this exploration not to box it. hopefully the proposed idea will get an interesting conversation sparked, and who knows, maybe even some cool projects in the near future :) enjoy! the ethereum project transmutation(s) liberty, when it begins to take root, is a plant of rapid growth. —george washington as does any successful idea, the ethereum project has metamorphosed many times and evolved into what it has become today. the growth happened in various ways while the available resources were channeled into strategic areas. in the beginning a big part of the focus was put into legal hacking and the creation of the required infrastructure to make the genesis sale happen. as we made our way through the legal bushes, getting to the point where we felt comfortable with the setup, the focus shifted towards the security and monitoring of the entire genesis sale operation. before turning the genesis sale switch on, we had in-depth security audits and created an entire ddos protection strategy with multiple layers and courses of action. as we approached the end of the genesis sale, as expected, the focus shifted mainly towards the development of the platform. after months of intense development, now the focus is shifting towards the immediate genesis launch, smoothing the rough edges and auditing the software in order to expose any security flaws that might have escaped us. soon, we will have ethereum released and all attention will go towards monitoring how it behaves “in the wild”. now the question is, what happens after it’s “out there”? what is the next phase of the project and what should we do now to prepare for it? the education aspect is regarded by everyone in the project as extremely important in both the short term and long term, especially since one of the points of this platform is the creation of a rich ecosystem of dapps, daos and other smart-contracts living on it. everyone agrees that in order to have a thriving ecosystem, we need to create favorable conditions for curious developers to play and experiment with ethereum. that also means providing the right tools, resources and tutorials – in short, making it as easy as possible for interested people to learn, collaborate and share knowledge. a proposed experiment: ethereum builders “in times of change, learners inherit the earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists.” —eric hoffer while exploring the idea behind educational portals (i.e. code academy) and researching into the effectiveness of different approaches, i stumbled upon a study on students taking online courses. according to this study, the students in massive open online courses are apt to take a passive approach to learning, avoiding collaboration with others, seeking only passing grades, and therefore not retaining new knowledge. even if in the beginning the students were motivated by acquiring new skills, as they progressed through the courses their focus shifted towards the grades and the completion of the course. following this rather disappointing result, the researchers suggested that when planning online courses, providers should develop engaging coursework that requires immediate peer interaction and use of new knowledge. this led me to the thinking that for something as radically new as ethereum, maybe looking for new approaches to education and collaboration could be key to unlocking the potential present in the already formed ethereum community. after experimenting with a few approaches, the idea behind the ethereum builders project started taking shape. as a project, it is formed from multiple components, but in this post i will focus only on the education area. in the current form, this is how things were thought: by combining screensharing, an open source upvoting interface and bitcoin, we create a way for the ethereum open source, knowledge-sharing economy to self-organize, self-sustain and distribute information towards the edges with maximum impact at virtually zero cost. special thanks goes to linda from code buddies for hacking this together and to aurel iancu for the invaluable help. because of lack of a better option, google hangouts with screensharing will be used as the interactive medium for the ethereum sessions. the web interface is an open source upvoting system, in this case used as a peer-to-peer self-organizing knowledge-sharing network. bitcoin (and soon eth) can serve as a tipping/donation mechanism for the course/hackathon organizers – tokens of appreciation for the invaluable shared knowledge. if the quality of the sessions is good, the screensharing hangouts could also be recorded and used for video tutorials and other educational content quite easily. for documentation and guides, gitbook seems like an interesting option. with it you can pull content directly from wiki pages that live on different repos (i.e. ethereum wiki, cpp wiki, go wiki, etc.) and display it in a nice web ui or ebook format. by pulling content from the wikis, whenever the documentation is updated the gitbook automatically updates as well. all in all it looks to be a good fit for what we need at the moment, since the documentation evolves at such a rapid pace in ethereum world. you can see the initial guide we hacked together here. as for badges and reputation, at the moment we will keep track of the organized sessions and community contributions in a spreadsheet. when the ethereum network goes live, we plan to transfer the contributions and reputation gained to the identity of user’s choice. this would then make it easier to: get recognition for the things you learn get/give recognition for the things you teach verify skills display and use your badges across the etherwebz (and not only) again, this just an experiment and we’ll be playing with a few of these things in the coming weeks to see what works and what doesn’t. so please be patient if something doesn’t work as expected and report it by opening an issue. if received well by the community, ethereum builders could unfold in a number of ways: multilingual ethereum study/hacking sessions for localized learning turning it into a dapp (proof of concept in the works) crowdfunding module(s) for projects, hackathons and workshops ethereum reputation system for the code whisperers, participants, builders, etc. asyncronous, self-organized global hackathons, breakathons and workshops community daofication ______________ insert here awesome idea(s)   fractals of knowledge-sharing and open collaboration you will only be remembered for two things: the problems you solve and the ones you create. —mike murdock in the current form, ethereum.builders is an open-ended tool that enables the ethereum community to coordinate, collaborate and share knowledge. the users will most likely discover new ways to use it on things that we haven't figured out yet. that being said, i'd like to walk through a few use cases in order to provide a starting point in this journey. exploring a few use cases: connecting ethereum builders with each other think of twitch, but for ethereum hacking. instead of watching other people playing games, you can watch people create awesome things on ethereum, and pick up new skills and knowledge along the way. from learning how to install an ethereum client on your machine to making your first dapp and showcasing it there’s an entire variety of interactions that can be explored. connecting ethereum builders with knowledge repositories organize the information and make it easily accessible. currently the ethereum knowledge base is fragmented between various repo wikis, forums and blog posts. as an alternative to this madness, i propose gitbook as a tool for organizing information and making it accessible in web, pdf, mobi and epub format. decentralized hackathons & breakathons we've dreamt about organizing a distributed global hackathon for a while now. now we have a way to do it. not only can we organize it, but at the same time, thanks to the hangouts on air feature, we can broadcast the event live on youtube. if you're interested in participating or sponsoring an event like this, please get in touch. breakathons are at the other end of the spectrum: instead of creating things, you try to break things. in our case we're talking about testing ethereum and seeing if anyone can break it. instead of being punished, the people able to break things will be rewarded, since thanks to their work ethereum grows stronger. you can think of it as a crowd security audit with prizes :) decentralized co-working sessions: the rise of adhocracy besides organizing hackathons, the hangouts can also serve as global co-working space(s). you can think of it like having a place where you can go and work on building cool things on ethereum together with other people from all around the world. the energy builds up when working together. blockchain research sessions: the academia connection use the available channels to reach the minds interested in researching the possibilities and problems that blockchain technology has. make it simple for these people to coordinate and exchange ideas in a highly interactive format. the rest is up to them. ethereum project updates besides the blog this could serve as an interactive channel of communication with the community and the developers swarming around the project. they’re not mutually exclusive tools but rather complementing each other. ethereum meet-ups sync physically with meetups.com decision to start to charging meet up organizers depending on the size of the group, some have already started looking for a replacement. online by design you remove the bottlenecks as in having "x" to organize a meetup call in a closed skype room with only a few select members. open the gates and allow the communities and their members to interact, get to know each other and synchronize. think of it as a meta layer that goes on top of physical meetups, allowing the local communities to become part of bigger communities (continental, worldwide, etc.). and now to wrap all these ideas into one, here is how it currently looks like: *some of the above sessions are just examples. feedback and help welcome for those who find this experiment interesting, have a question or want to help, you can see it in action at ethereum.builders or join us for a chat on gitter. if you spot a bug or something that’s not working properly please open an issue and if you want to contribute code or create bounties for certain features please drop by our repo ^_^ the experiment is on! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle the three transitions 2023 jun 09 see all posts special thanks to dan finlay, karl floersch, david hoffman, and the scroll and soulwallet teams for feedback and review and suggestions. as ethereum transitions from a young experimental technology into a mature tech stack that is capable of actually bringing an open, global and permissionless experience to average users, there are three major technical transitions that the stack needs to undergo, roughly simultaneously: the l2 scaling transition everyone moving to rollups the wallet security transition everyone moving to smart contract wallets the privacy transition making sure privacy-preserving funds transfers are available, and making sure all of the other gadgets that are being developed (social recovery, identity, reputation) are privacy-preserving the ecosystem transition triangle. you can only pick 3 out of 3. without the first, ethereum fails because each transaction costs $3.75 ($82.48 if we have another bull run), and every product aiming for the mass market inevitably forgets about the chain and adopts centralized workarounds for everything. without the second, ethereum fails because users are uncomfortable storing their funds (and non-financial assets), and everyone moves onto centralized exchanges. without the third, ethereum fails because having all transactions (and poaps, etc) available publicly for literally anyone to see is far too high a privacy sacrifice for many users, and everyone moves onto centralized solutions that at least somewhat hide your data. these three transitions are crucial for the reasons above. but they are also challenging because of the intense coordination involved to properly resolve them. it's not just features of the protocol that need to improve; in some cases, the way that we interact with ethereum needs to change pretty fundamentally, requiring deep changes from applications and wallets. the three transitions will radically reshape the relationship between users and addresses in an l2 scaling world, users are going to exist on lots of l2s. are you a member of exampledao, which lives on optimism? then you have an account on optimism! are you holding a cdp in a stablecoin system on zksync? then you have an account on zksync! did you once go try some application that happened to live on kakarot? then you have an account on kakarot! the days of a user having only one address will be gone. i have eth in four places, according to my brave wallet view. and yes, arbitrum and arbitrum nova are different. don't worry, it will get more confusing over time! smart contract wallets add more complexity, by making it much more difficult to have the same address across l1 and the various l2s. today, most users are using externally owned accounts, whose address is literally a hash of the public key that is used to verify signatures so nothing changes between l1 and l2. with smart contract wallets, however, keeping one address becomes more difficult. although a lot of work has been done to try to make addresses be hashes of code that can be equivalent across networks, most notably create2 and the erc-2470 singleton factory, it's difficult to make this work perfectly. some l2s (eg. "type 4 zk-evms") are not quite evm equivalent, often using solidity or an intermediate assembly instead, preventing hash equivalence. and even when you can have hash equivalence, the possibility of wallets changing ownership through key changes creates other unintuitive consequences. privacy requires each user to have even more addresses, and may even change what kinds of addresses we're dealing with. if stealth address proposals become widely used, instead of each user having only a few addresses, or one address per l2, users might have one address per transaction. other privacy schemes, even existing ones such as tornado cash, change how assets are stored in a different way: many users' funds are stored in the same smart contract (and hence at the same address). to send funds to a specific user, users will need to rely on the privacy scheme's own internal addressing system. as we've seen, each of the three transitions weaken the "one user ~= one address" mental model in different ways, and some of these effects feed back into the complexity of executing the transitions. two particular points of complexity are: if you want to pay someone, how will you get the information on how to pay them? if users have many assets stored in different places across different chains, how do they do key changes and social recovery? the three transitions and on-chain payments (and identity) i have coins on scroll, and i want to pay for coffee (if the "i" is literally me, the writer of this article, then "coffee" is of course a metonymy for "green tea"). you are selling me the coffee, but you are only set up to receive coins on taiko. wat do? there are basically two solutions: receiving wallets (which could be merchants, but also could just be regular individuals) try really hard to support every l2, and have some automated functionality for consolidating funds asynchronously. the recipient provides their l2 alongside their address, and the sender's wallet automatically routes funds to the destination l2 through some cross-l2 bridging system. of course, these solutions can be combined: the recipient provides the list of l2s they're willing to accept, and the sender's wallet figures out payment, which could involve either a direct send if they're lucky, or otherwise a cross-l2 bridging path. but this is only one example of a key challenge that the three transitions introduce: simple actions like paying someone start to require a lot more information than just a 20-byte address. a transition to smart contract wallets is fortunately not a large burden on the addressing system, but there are still some technical issues in other parts of the application stack that need to be worked through. wallets will need to be updated to make sure that they do not send only 21000 gas along with a transaction, and it will be even more important to ensure that the payment receiving side of a wallet tracks not only eth transfers from eoas, but also eth sent by smart contract code. apps that rely on the assumption that address ownership is immutable (eg. nfts that ban smart contracts to enforce royalties) will have to find other ways of achieving their goals. smart contract wallets will also make some things easier notably, if someone receives only a non-eth erc20 token, they will be able to use erc-4337 paymasters to pay for gas with that token. privacy, on the other hand, once again poses major challenges that we have not really dealt with yet. the original tornado cash did not introduce any of these issues, because it did not support internal transfers: users could only deposit into the system and withdraw out of it. once you can make internal transfers, however, users will need to use the internal addressing scheme of the privacy system. in practice, a user's "payment information" would need to contain both (i) some kind of "spending pubkey", a commitment to a secret that the recipient could use to spend, and (ii) some way for the sender to send encrypted information that only the recipient can decrypt, to help the recipient discover the payment. stealth address protocols rely on a concept of meta-addresses, which work in this way: one part of the meta-address is a blinded version of the sender's spending key, and another part is the sender's encryption key (though a minimal implementation could set those two keys to be the same). schematic overview of an abstract stealth address scheme based on encryption and zk-snarks. a key lesson here is that in a privacy-friendly ecosystem, a user will have both spending pubkeys and encryption pubkeys, and a user's "payment information" will have to include both keys. there are also good reasons other than payments to expand in this direction. for example, if we want ethereum-based encrypted email, users will need to publicly provide some kind of encryption key. in "eoa world", we could re-use account keys for this, but in a safe smart-contract-wallet world, we probably should have more explicit functionality for this. this would also help in making ethereum-based identity more compatible with non-ethereum decentralized privacy ecosystems, most notably pgp keys. the three transitions and key recovery the default way to implement key changes and social recovery in a many-address-per-user world is to simply have users run the recovery procedure on each address separately. this can be done in one click: the wallet can include software to execute the recovery procedure across all of a user's addresses at the same time. however, even with such ux simplifications, naive multi-address recovery has three issues: gas cost impracticality: this one is self-explanatory. counterfactual addresses: addresses for which the smart contract has not yet been published (in practice, this will mean an account that you have not yet sent funds from). you as a user have a potentially unlimited number of counterfactual addresses: one or more on every l2, including l2s that do not yet exist, and a whole other infinite set of counterfactual addresses arising from stealth address schemes. privacy: if a user intentionally has many addresses to avoid linking them to each other, they certainly do not want to publicly link all of them by recovering them at or around the same time! solving these problems is hard. fortunately, there is a somewhat elegant solution that performs reasonably well: an architecture that separates verification logic and asset holdings. each user has a keystore contract, which exists in one location (could either be mainnet or a specific l2). users then have addresses on different l2s, where the verification logic of each of those addresses is a pointer to the keystore contract. spending from those addresses would require a proof going into the keystore contract showing the current (or, more realistically, very recent) spending public key. the proof could be implemented in a few ways: direct read-only l1 access inside the l2. it's possible to modify l2s to give them a way to directly read l1 state. if the keystore contract is on l1, this would mean that contracts inside l2 can access the keystore "for free" merkle branches. merkle branches can prove l1 state to an l2, or l2 state to an l1, or you can combine the two to prove parts of the state of one l2 to another l2. the main weakness of merkle proofs is high gas costs due to proof length: potentially 5 kb for a proof, though this will reduce to < 1 kb in the future due to verkle trees. zk-snarks. you can reduce data costs by using a zk-snark of a merkle branch instead of the branch itself. it's possible to build off-chain aggregation techniques (eg. on top of eip-4337) to have one single zk-snark verify all cross-chain state proofs in a block. kzg commitments. either l2s, or schemes built on top of them, could introduce a sequential addressing system, allowing proofs of state inside this system to be a mere 48 bytes long. like with zk-snarks, a multiproof scheme could merge all of these proofs into a single proof per block. if we want to avoid making one proof per transaction, we can implement a lighter scheme that only requires a cross-l2 proof for recovery. spending from an account would depend on a spending key whose corresponding pubkey is stored within that account, but recovery would require a transaction that copies over the current spending_pubkey in the keystore. funds in counterfactual addresses are safe even if your old keys are not: "activating" a counterfactual address to turn it into a working contract would require making a cross-l2 proof to copy over the current spending_pubkey. this thread on the safe forums describes how a similar architecture might work. to add privacy to such a scheme, then we just encrypt the pointer, and we do all of our proving inside zk-snarks: with more work (eg. using this work as a starting point), we could also strip out most of the complexity of zk-snarks and make a more bare-bones kzg-based scheme. these schemes can get complex. on the plus side, there are many potential synergies between them. for example, the concept of "keystore contracts" could also be a solution to the challenge of "addresses" mentioned in the previous section: if we want users to have persistent addresses, that do not change every time the user updates a key, we could put stealth meta-addresses, encryption keys, and other information into the keystore contract, and use the address of the keystore contract as a user's "address". lots of secondary infrastructure needs to update using ens is expensive. today, in june 2023, the situation is not too bad: the transaction fee is significant, but it's still comparable to the ens domain fee. registering zuzalu.eth cost me roughly $27, of which $11 was transaction fees. but if we have another bull market, fees will skyrocket. even without eth price increases, gas fees returning to 200 gwei would raise the tx fee of a domain registration to $104. and so if we want people to actually use ens, especially for use cases like decentralized social media where users demand nearly-free registration (and the ens domain fee is not an issue because these platforms offer their users sub-domains), we need ens to work on l2. fortunately, the ens team has stepped up, and ens on l2 is actually happening! erc-3668 (aka "the ccip standard"), together with ensip-10, provide a way to have ens subdomains on any l2 automatically be verifiable. the ccip standard requires setting up a smart contract that describes a method for verifying proofs of data on l2, and a domain (eg. optinames uses ecc.eth) can be put under the control of such a contract. once the ccip contract controls ecc.eth on l1, accessing some subdomain.ecc.eth will automatically involve finding and verifying a proof (eg. merkle branch) of the state in l2 that actually stores that particular subdomain. actually fetching the proofs involves going to a list of urls stored in the contract, which admittedly feels like centralization, though i would argue it really isn't: it's a 1-of-n trust model (invalid proofs get caught by the verification logic in the ccip contract's callback function, and as long as even one of the urls returns a valid proof, you're good). the list of urls could contain dozens of them. the ens ccip effort is a success story, and it should be viewed as a sign that radical reforms of the kind that we need are actually possible. but there's a lot more application-layer reform that will need to be done. a few examples: lots of dapps depend on users providing off-chain signatures. with externally-owned accounts (eoas), this is easy. erc-1271 provides a standardized way to do this for smart contract wallets. however, lots of dapps still don't support erc-1271; they will need to. dapps that use "is this an eoa?" to discriminate between users and contracts (eg. to prevent transfer or enforce royalties) will break. in general, i advise against attempting to find a purely technical solution here; figuring out whether or not a particular transfer of cryptographic control is a transfer of beneficial ownership is a difficult problem and probably not solvable without resolving to some off-chain community-driven mechanisms. most likely, applications will have to rely less on preventing transfers and more on techniques like harberger taxes. how wallets interact with spending and encryption keys will have to be improved. currently, wallets often use deterministic signatures to generate application-specific keys: signing a standard nonce (eg. the hash of the application's name) with an eoa's private key generates a deterministic value that cannot be generated without the private key, and so it's secure in a purely technical sense. however, these techniques are "opaque" to the wallet, preventing the wallet from implementing user-interface level security checks. in a more mature ecosystem, signing, encryption and related functionalities will have to be handled by wallets more explicitly. light clients (eg. helios) will have to verify l2s and not just the l1. today, light clients focus on checking the validity of the l1 headers (using the light client sync protocol), and verifying merkle branches of l1 state and transactions rooted in the l1 header. tomorrow, they will also need to verify a proof of l2 state rooted in the state root stored in the l1 (a more advanced version of this would actually look at l2 pre-confirmations). wallets will need to secure both assets and data today, wallets are in the business of securing assets. everything lives on-chain, and the only thing that the wallet needs to protect is the private key that is currently guarding those assets. if you change the key, you can safely publish your previous private key on the internet the next day. in a zk world, however, this is no longer true: the wallet is not just protecting authentication credentials, it's also holding your data. we saw the first signs of such a world with zupass, the zk-snark-based identity system that was used at zuzalu. users had a private key that they used to authenticate to the system, which could be used to make basic proofs like "prove i'm a zuzalu resident, without revealing which one". but the zupass system also began to have other apps built on top, most notably stamps (zupass's version of poaps). one of my many zupass stamps, confirming that i am a proud member of team cat. the key feature that stamps offer over poaps is that stamps are private: you hold the data locally, and you only zk-prove a stamp (or some computation over the stamps) to someone if you want them to have that information about you. but this creates added risk: if you lose that information, you lose your stamps. of course, the problem of holding data can be reduced to the problem of holding a single encryption key: some third party (or even the chain) can hold an encrypted copy of the data. this has the convenient advantage that actions you take don't change the encryption key, and so do not require any interactions with the system holding your encryption key safe. but even still, if you lose your encryption key, you lose everything. and on the flip side, if someone sees your encryption key, they see everything that was encrypted to that key. zupass's de-facto solution was to encourage people to store their key on multiple devices (eg. laptop and phone), as the chance that they would lose access to all devices at the same time is tiny. we could go further, and use secret sharing to store the key, split between multiple guardians. this kind of social recovery via mpc is not a sufficient solution for wallets, because it means that not only current guardians but also previous guardians could collude to steal your assets, which is an unacceptably high risk. but privacy leaks are generally a lower risk than total asset loss, and someone with a high-privacy-demanding use case could always accept a higher risk of loss by not backing up the key associated with those privacy-demanding actions. to avoid overwheming the user with a byzantine system of multiple recovery paths, wallets that support social recovery will likely need to manage both recovery of assets and recovery of encryption keys. back to identity one of the common threads of these changes is that the concept of an "address", a cryptographic identifier that you use to represent "you" on-chain, will have to radically change. "instructions for how to interact with me" would no longer just be an eth address; they would have to be, in some form, some combination of multiple addresses on multiple l2s, stealth meta-addresses, encryption keys, and other data. one way to do this is to make ens your identity: your ens record could just contain all of this information, and if you send someone bob.eth (or bob.ecc.eth, or...), they could look up and see everything about how to pay and interact with you, including in the more complicated cross-domain and privacy-preserving ways. but this ens-centric approach has two weaknesses: it ties too many things to your name. your name is not you, your name is one of many attributes of you. it should be possible to change your name without moving over your entire identity profile and updating a whole bunch of records across many applications. you can't have trustless counterfactual names. one key ux feature of any blockchain is the ability to send coins to people who have not interacted with the chain yet. without such a functionality, there is a catch-22: interacting with the chain requires paying transaction fees, which requires... already having coins. eth addresses, including smart contract addresses with create2, have this feature. ens names don't, because if two bobs both decide off-chain that they are bob.ecc.eth, there's no way to choose which one of them gets the name. one possible solution is to put more things into the keystore contract mentioned in the architecture earlier in this post. the keystore contract could contain all of the various information about you and how to interact with you (and with ccip, some of that info could be off-chain), and users would use their keystore contract as their primary identifier. but the actual assets that they receive would be stored in all kinds of different places. keystore contracts are not tied to a name, and they are counterfactual-friendly: you can generate an address that can provably only be initialized by a keystore contract that has certain fixed initial parameters. another category of solutions has to do with abandoning the concept of user-facing addresses altogether, in a similar spirit to the bitcoin payment protocol. one idea is to rely more heavily on direct communication channels between the sender and the recipient; for example, the sender could send a claim link (either as an explicit url or a qr code) which the recipient could use to accept the payment however they wish. regardless of whether the sender or the recipient acts first, greater reliance on wallets directly generating up-to-date payment information in real time could reduce friction. that said, persistent identifiers are convenient (especially with ens), and the assumption of direct communication between sender and recipient is a really tricky one in practice, and so we may end up seeing a combination of different techniques. in all of these designs, keeping things both decentralized and understandable to users is paramount. we need to make sure that users have easy access to an up-to-date view of what their current assets are and what messages have been published that are intended for them. these views should depend on open tools, not proprietary solutions. it will take hard work to avoid the greater complexity of payment infrastructure from turning into an opaque "tower of abstraction" where developers have a hard time making sense of what's going on and adapting it to new contexts. despite the challenges, achieving scalability, wallet security, and privacy for regular users is crucial for ethereum's future. it is not just about technical feasibility but about actual accessibility for regular users. we need to rise to meet this challenge. dark mode toggle my 40-liter backpack travel guide 2022 jun 20 see all posts special thanks to liam horne for feedback and review. i received no money from and have never even met any of the companies making the stuff i'm shilling here (with the sole exception of unisocks); this is all just an honest listing of what works for me today. i have lived as a nomad for the last nine years, taking 360 flights travelling over 1.5 million kilometers (assuming flight paths are straight, ignoring layovers) during that time. during this time, i've considerably optimized the luggage i carry along with me: from a 60-liter shoulder bag with a separate laptop bag, to a 60-liter shoulder bag that can contain the laptop bag, and now to a 40-liter packpage that can contain the laptop bag along with all the supplies i need to live my life. the purpose of this post will be to go through the contents, as well as some of the tips that i've learned for how you too can optimize your travel life and never have to wait at a luggage counter again. there is no obligation to follow this guide in its entirety; if you have important needs that differ from mine, you can still get a lot of the benefits by going a hybrid route, and i will talk about these options too. this guide is focused on my own experiences; plenty of other people have made their own guides and you should look at them too. /r/onebag is an excellent subreddit for this. the backpack, with the various sub-bags laid out separately. yes, this all fits in the backpack, and without that much effort to pack and unpack. as a point of high-level organization, notice the bag-inside-a-bag structure. i have a t-shirt bag, an underwear bag, a sock bag, a toiletries bag, a dirty-laundry bag, a medicine bag, a laptop bag, and various small bags inside the inner compartment of my backpack, which all fit into a 40-liter hynes eagle backpack. this structure makes it easy to keep things organized. it's like frugality, but for cm3 instead of dollars the general principle that you are trying to follow is that you're trying to stay within a "budget" while still making sure you have everything that you need much like normal financial planning of the type that almost everyone, with the important exception of crypto participants during bull runs, is used to dealing with. a key difference here is that instead of optimizing for dollars, you're optimizing for cubic centimeters. of course, none of the things that i recommend here are going to be particularly hard on your dollars either, but minimizing cm3 is the primary objective. what do i mean by this? well, i mean getting items like this: electric shaver. about 5cm long and 2.5cm wide at the top. no charger or handle is required: it's usbc pluggable, your phone is the charger and handle. buy on amazon here (told you it's not hard on your dollars!) and this: charger for mobile phone and laptop (can charge both at the same time)! about 5x5x2.5 cm. buy here. and there's more. electric toothbrushes are normally known for being wide and bulky. but they don't have to be! here is an electric toothbrush that is rechargeable, usbc-friendly (so no extra charging equipment required), only slightly wider than a regular toothbrush, and costs about $30, plus a couple dollars every few months for replacement brush heads. for connecting to various different continents' plugs, you can either use any regular reasonably small universal adapter, or get the zendure passport iii which combines a universal adapter with a charger, so you can plug in usbc cables to charge your laptop and multiple other devices directly (!!). as you might have noticed, a key ingredient in making this work is to be a usbc maximalist. you should strive to ensure that every single thing you buy is usbc-friendly. your laptop, your phone, your toothbrush, everything. this ensures that you don't need to carry any extra equipment beyond one charger and 1-2 charging cables. in the last ~3 years, it has become much easier to live the usbc maximalist life; enjoy it! be a uniqlo maximalist for clothing, you have to navigate a tough tradeoff between price, cm3 and the clothing looking reasonably good. fortunately, many of the more modern brands do a great job of fulfilling all three at the same time! my current strategy is to be a uniqlo maximalist: altogether, about 70% of the clothing items in my bag are from uniqlo. this includes: 8 t-shirts, of which 6 are this type from uniqlo 8 pairs of underwear, mostly various uniqlo products 8 socks, of which none are uniqlo (i'm less confident about what to do with socks than with other clothing items, more on this later) heat-tech tights, from uniqlo heat-tech sweater, from uniqlo packable jacket, from uniqlo shorts that also double as a swimsuit, from.... ok fine, it's also uniqlo. there are other stores that can give you often equally good products, but uniqlo is easily accessible in many (though not all) of the regions i visit and does a good job, so i usually just start and stop there. socks socks are a complicated balancing act between multiple desired traits: low cm3 easy to put on warm (when needed) comfortable the ideal scenario is if you find low-cut or ankle socks comfortable to wear, and you never go to cold climates. these are very low on cm3, so you can just buy those and be happy. but this doesn't work for me: i sometimes visit cold areas, i don't find ankle socks comfortable and prefer something a bit longer, and i need to be comfortable for my long runs. furthermore, my large foot size means that uniqlo's one-size-fits-all approach does not work well for me: though i can put the socks on, it often takes a long time to do so (especially after a shower), and the socks rip often. so i've been exploring various brands to try to find a solution (recently trying cep and darntough). i generally try to find socks that cover the ankle but don't go much higher than that, and i have one pair of long ones for when i go to the snowier places. my sock bag is currently larger than my underwear bag, and only a bit smaller than my t-shirt bag: both a sign of the challenge of finding good socks, and a testament to uniqlo's amazing airism t-shirts. once you do find a pair of socks that you like, ideally you should just buy many copies of the same type. this removes the effort of searching for a matching pair in your bag, and it ensures that if one of your socks rips you don't have to choose between losing the whole pair and wearing mismatched socks. for shoes, you probably want to limit yourself to at most two: some heavier shoes that you can just wear, and some very cm3-light alternative, such as flip-flops. layers there is a key mathematical reason why dressing in layers is a good idea: it lets you cover many possible temperature ranges with fewer clothing items. temperature (°c) clothing 20° 13° + 7° + 0° + + you want to keep the t-shirt on in all cases, to protect the other layers from getting dirty. but aside from that, the general rule is: if you choose n clothing items, with levels of warmness spread out across powers of two, then you can be comfortable in \(2^n\) different temperature ranges by binary-encoding the expected temperature in the clothing you wear. for not-so-cold climates, two layers (sweater and jacket) are fine. for a more universal range of climates you'll want three layers: light sweater, heavy sweater and heavy jacket, which can cover \(2^3 = 8\) different temperature ranges all the way from summer to siberian winter (of course, heavy winter jackets are not easily packable, so you may have to just wear it when you get on the plane). this layering principle applies not just to upper-wear, but also pants. i have a pair of thin pants plus uniqlo tights, and i can wear the thin pants alone in warmer climates and put the uniqlo tights under them in colder climates. the tights also double as pyjamas. my miscellaneous stuff the internet constantly yells at me for not having a good microphone. i solved this problem by getting a portable microphone! my workstation, using the apogee hypemic travel microphone (unfortunately micro-usb, not usbc). a toilet paper roll works great as a stand, but i've also found that having a stand is not really necessary and you can just let the microphone lie down beside your laptop. next, my laptop stand. laptop stands are great for improving your posture. i have two recommendations for laptop stands, one medium-effective but very light on cm3, and one very effective but heavier on cm3. the lighter one: majextand the more powerful one: nexstand nexstand is the one in the picture above. majextand is the one glued to the bottom of my laptop now: i have used both, and recommend both. in addition to this i also have another piece of laptop gear: a 20000 mah laptop-friendly power bank. this adds even more to my laptop's already decent battery life, and makes it generally easy to live on the road. now, my medicine bag: this contains a combination of various life-extension medicines (metformin, ashwagandha, and some vitamins), and covid defense gear: a co2 meter (co2 concentration minus 420 roughly gives you how much human-breathed-out air you're breathing in, so it's a good proxy for virus risk), masks, antigen tests and fluvoxamine. the tests were a free care package from the singapore government, and they happened to be excellent on cm3 so i carry them around. covid defense and life extension are both fields where the science is rapidly evolving, so don't blindly follow this static list; follow the science yourself or listen to the latest advice of an expert that you do trust. air filters and far-uvc (especially 222 nm) lamps are also promising covid defense options, and portable versions exist for both. at this particular time i don't happen to have a first aid kit with me, but in general it's also recommended; plenty of good travel options exist, eg. this. finally, mobile data. generally, you want to make sure you have a phone that supports esim. these days, more and more phones do. wherever you go, you can buy an esim for that place online. i personally use airalo, but there are many options. if you are lazy, you can also just use google fi, though in my experience google fi's quality and reliability of service tends to be fairly mediocre. have some fun! not everything that you have needs to be designed around cm3 minimization. for me personally, i have four items that are not particularly cm3 optimized but that i still really enjoy having around. my laptop bag, bought in an outdoor market in zambia. unisocks. sweatpants for indoor use, that are either fox-themed or shiba inu-themed depending on whom you ask. gloves (phone-friendly): i bought the left one for $4 in mong kok and the right one for $5 in chinatown, toronto back in 2016. by coincidence, i lost different ones from each pair, so the remaining two match. i keep them around as a reminder of the time when money was much more scarce for me. the more you save space on the boring stuff, the more you can leave some space for a few special items that can bring the most joy to your life. how to stay sane as a nomad many people find the nomad lifestyle to be disorienting, and report feeling comfort from having a "permanent base". i find myself not really having these feelings: i do feel disorientation when i change locations more than once every ~7 days, but as long as i'm in the same place for longer than that, i acclimate and it "feels like home". i can't tell how much of this is my unique difficult-to-replicate personality traits, and how much can be done by anyone. in general, some tips that i recommend are: plan ahead: make sure you know where you'll be at least a few days in advance, and know where you're going to go when you land. this reduces feelings of uncertainty. have some other regular routine: for me, it's as simple as having a piece of dark chocolate and a cup of tea every morning (i prefer bigelow green tea decaf, specifically the 40-packs, both because it's the most delicious decaf green tea i've tried and because it's packaged in a four-teabag-per-bigger-bag format that makes it very convenient and at the same time cm3-friendly). having some part of your lifestyle the same every day helps me feel grounded. the more digital your life is, the more you get this "for free" because you're staring into the same computer no matter what physical location you're in, though this does come at the cost of nomadding potentially providing fewer benefits. your nomadding should be embedded in some community: if you're just being a lowest-common-denominator tourist, you're doing it wrong. find people in the places you visit who have some key common interest (for me, of course, it's blockchains). make friends in different cities. this helps you learn about the places you visit and gives you an understanding of the local culture in a way that "ooh look at the 800 year old statue of the emperor" never will. finally, find other nomad friends, and make sure to intersect with them regularly. if home can't be a single place, home can be the people you jump places with. have some semi-regular bases: you don't have to keep visiting a completely new location every time. visiting a place that you have seen before reduces mental effort and adds to the feeling of regularity, and having places that you visit frequently gives you opportunities to put stuff down, and is important if you want your friendships and local cultural connections to actually develop. how to compromise not everyone can survive with just the items i have. you might have some need for heavier clothing that cannot fit inside one backpack. you might be a big nerd in some physical-stuff-dependent field: i know life extension nerds, covid defense nerds, and many more. you might really love your three monitors and keyboard. you might have children. the 40-liter backpack is in my opinion a truly ideal size if you can manage it: 40 liters lets you carry a week's worth of stuff, and generally all of life's basic necessities, and it's at the same time very carry-friendly: i have never had it rejected from carry-on in all the flights on many kinds of airplane that i have taken it, and when needed i can just barely stuff it under the seat in front of me in a way that looks legit to staff. once you start going lower than 40 liters, the disadvantages start stacking up and exceeding the marginal upsides. but if 40 liters is not enough for you, there are two natural fallback options: a larger-than-40 liter backpack. you can find 50 liter backpacks, 60 liter backpacks or even larger (i highly recommend backpacks over shoulder bags for carrying friendliness). but the higher you go, the more tiring it is to carry, the more risk there is on your spine, and the more you incur the risk that you'll have a difficult situation bringing it as a carry-on on the plane and might even have to check it. backpack plus mini-suitcase. there are plenty of carry-on suitcases that you can buy. you can often make it onto a plane with a backpack and a mini-suitcase. this depends on you: you may find this to be an easier-to-carry option than a really big backpack. that said, there is sometimes a risk that you'll have a hard time carrying it on (eg. if the plane is very full) and occasionally you'll have to check something. either option can get you up to a respectable 80 liters, and still preserve a lot of the benefits of the 40-liter backpack lifestyle. backpack plus mini-suitcase generally seems to be more popular than the big backpack route. it's up to you to decide which tradeoffs to take, and where your personal values lie! dark mode toggle having a safe cex: proof of solvency and beyond 2022 nov 19 see all posts special thanks to balaji srinivasan, and coinbase, kraken and binance staff for discussion. every time a major centralized exchange blows up, a common question that comes up is whether or not we can use cryptographic techniques to solve the problem. rather than relying solely on "fiat" methods like government licenses, auditors and examining the corporate governance and the backgrounds of the individuals running the exchange, exchanges could create cryptographic proofs that show that the funds they hold on-chain are enough to cover their liabilities to their users. even more ambitiously, an exchange could build a system where it can't withdraw a depositor's funds at all without their consent. potentially, we could explore the entire spectrum between the "don't be evil" aspiring-good-guy cex and the "can't be evil", but for-now inefficient and privacy-leaking, on-chain dex. this post will get into the history of attempts to move exchanges one or two steps closer to trustlessness, the limitations of these techniques, and some newer and more powerful ideas that rely on zk-snarks and other advanced technologies. balance lists and merkle trees: old-school proof-of-solvency the earliest attempts by exchanges to try to cryptographically prove that they are not cheating their users go back quite far. in 2011, then-largest bitcoin exchange mtgox proved that they had funds by sending a transaction that moved 424242 btc to a pre-announced address. in 2013, discussions started on how to solve the other side of the problem: proving the total size of customers' deposits. if you prove that customers' deposits equal x ("proof of liabilities"), and prove ownership of the private keys of x coins ("proof of assets"), then you have a proof of solvency: you've proven the exchange has the funds to pay back all of its depositors. the simplest way to prove deposits is to simply publish a list of (username, balance) pairs. each user can check that their balance is included in the list, and anyone can check the full list to see that (i) every balance is non-negative, and (ii) the total sum is the claimed amount. of course, this breaks privacy, so we can change the scheme a little bit: publish a list of (hash(username, salt), balance) pairs, and send each user privately their salt value. but even this leaks balances, and it leaks the pattern of changes in balances. the desire to preserve privacy brings us to the next invention: the merkle tree technique. green: charlie's node. blue: nodes charlie will receive as part of his proof. yellow: root node, publicly shown to everyone. the merkle tree technique consists of putting the table of customers' balances into a merkle sum tree. in a merkle sum tree, each node is a (balance, hash) pair. the bottom-layer leaf nodes represent the balances and salted username hashes of individual customers. in each higher-layer node, the balance is the sum of the two balances below, and the hash is the hash of the two nodes below. a merkle sum proof, like a merkle proof, is a "branch" of the tree, consisting of the sister nodes along the path from a leaf to the root. the exchange would send each user a merkle sum proof of their balance. the user would then have a guarantee that their balance is correctly included as part of the total. a simple example code implementation can be found here. # the function for computing a parent node given two child nodes def combine_tree_nodes(l, r): l_hash, l_balance = l r_hash, r_balance = r assert l_balance >= 0 and r_balance >= 0 new_node_hash = hash( l_hash + l_balance.to_bytes(32, 'big') + r_hash + r_balance.to_bytes(32, 'big') ) return (new_node_hash, l_balance + r_balance) # builds a full merkle tree. stored in flattened form where # node i is the parent of nodes 2i and 2i+1 def build_merkle_sum_tree(user_table: "list[(username, salt, balance)]"): tree_size = get_next_power_of_2(len(user_table)) tree = ( [none] * tree_size + [userdata_to_leaf(*user) for user in user_table] + [empty_leaf for _ in range(tree_size len(user_table))] ) for i in range(tree_size 1, 0, -1): tree[i] = combine_tree_nodes(tree[i*2], tree[i*2+1]) return tree # root of a tree is stored at index 1 in the flattened form def get_root(tree): return tree[1] # gets a proof for a node at a particular index def get_proof(tree, index): branch_length = log2(len(tree)) 1 # ^ = bitwise xor, x ^ 1 = sister node of x index_in_tree = index + len(tree) // 2 return [tree[(index_in_tree // 2**i) ^ 1] for i in range(branch_length)] # verifies a proof (duh) def verify_proof(username, salt, balance, index, user_table_size, root, proof): leaf = userdata_to_leaf(username, salt, balance) branch_length = log2(get_next_power_of_2(user_table_size)) 1 for i in range(branch_length): if index & (2**i): leaf = combine_tree_nodes(proof[i], leaf) else: leaf = combine_tree_nodes(leaf, proof[i]) return leaf == root privacy leakage in this design is much lower than with a fully public list, and it can be decreased further by shuffling the branches each time a root is published, but some privacy leakage is still there: charlie learns that someone has a balance of 164 eth, some two users have balances that add up to 70 eth, etc. an attacker that controls many accounts could still potentially learn a significant amount about the exchange's users. one important subtlety of the scheme is the possibility of negative balances: what if an exchange that has 1390 eth of customer balances but only 890 eth in reserves tries to make up the difference by adding a -500 eth balance under a fake account somewhere in the tree? it turns out that this possibility does not break the scheme, though this is the reason why we specifically need a merkle sum tree and not a regular merkle tree. suppose that henry is the fake account controlled by the exchange, and the exchange puts -500 eth there: greta's proof verification would fail: the exchange would have to give her henry's -500 eth node, which she would reject as invalid. eve and fred's proof verification would also fail, because the intermediate node above henry has -230 total eth, and so is also invalid! to get away with the theft, the exchange would have to hope that nobody in the entire right half of the tree checks their balance proof. if the exchange can identify 500 eth worth of users that they are confident will either not bother to check the proof, or will not be believed when they complain that they never received a proof, they could get away with the theft. but then the exchange could also just exclude those users from the tree and have the same effect. hence, the merkle tree technique is basically as good as a proof-of-liabilities scheme can be, if only achieving a proof of liabilities is the goal. but its privacy properties are still not ideal. you can go a little bit further by using merkle trees in more clever ways, like making each satoshi or wei a separate leaf, but ultimately with more modern tech there are even better ways to do it. improving privacy and robustness with zk-snarks zk-snarks are a powerful technology. zk-snarks may be to cryptography what transformers are to ai: a general-purpose technology that is so powerful that it will completely steamroll a whole bunch of application-specific techniques for a whole bunch of problems that were developed in the decades prior. and so, of course, we can use zk-snarks to greatly simplify and improve privacy in proof-of-liabilities protocols. the simplest thing that we can do is put all users' deposits into a merkle tree (or, even simpler, a kzg commitment), and use a zk-snark to prove that all balances in the tree are non-negative and add up to some claimed value. if we add a layer of hashing for privacy, the merkle branch (or kzg proof) given to each user would reveal nothing about the balance of any other user. using kzg commitments is one way to avoid privacy leakage, as there is no need to provide "sister nodes" as proofs, and a simple zk-snark can be used to prove the sum of the balances and that each balance is non-negative. we can prove the sum and non-negativity of balances in the above kzg with a special-purpose zk-snark. here is one simple example way to do this. we introduce an auxiliary polynomial \(i(x)\), which "builds up the bits" of each balance (we assume for the sake of example that balances are under \(2^{15}\)) and where every 16th position tracks a running total with an offset so that it sums to zero only if the actual total matches the declared total. if \(z\) is an order-128 root of unity, we might prove the equations: \(i(z^{16x}) = 0\) \(i(z^{16x + 14}) = p(\omega^{2x+1})\) \(i(z^{i}) 2*i(z^{i-1}) \in \{0, 1\}\ \ if\ \ i\ \ mod\ 16 \not \in \{0, 15\}\) \(i(z^{16*x + 15}) = i(z^{16*x-1}) + i(z^{16*x+14}) \frac{the\ declared\ total}{user\ count}\) the first values of a valid setting for \(i(x)\) would be 0 0 0 0 0 0 0 0 0 0 1 2 5 10 20 -165 0 0 0 0 0 0 0 0 0 1 3 6 12 25 50 -300 ... see here and here in my post on zk-snarks for further explanation of how to convert equations like these into a polynomial check and then into a zk-snark. this isn't an optimal protocol, but it does show how these days these kinds of cryptographic proofs are not that spooky! with only a few extra equations, constraint systems like this can be adapted to more complex settings. for example, in a leverage trading system, an individual users having negative balances is acceptable but only if they have enough other assets to cover the funds with some collateralization margin. a snark could be used to prove this more complicated constraint, reassuring users that the exchange is not risking their funds by secretly exempting other users from the rules. in the longer-term future, this kind of zk proof of liabilities could perhaps be used not just for customer deposits at exchanges, but for lending more broadly. anyone taking out a loan would put a record into a polynomial or a tree containing that loan, and the root of that structure would get published on-chain. this would let anyone seeking a loan zk-prove to the lender that they have not yet taken out too many other loans. eventually, legal innovation could even make loans that have been committed to in this way higher-priority than loans that have not. this leads us in exactly the same direction as one of the ideas that was discussed in the "decentralized society: finding web3's soul" paper: a general notion of negative reputation or encumberments on-chain through some form of "soulbound tokens". proof of assets the simplest version of proof of assets is the protocol that we saw above: to prove that you hold x coins, you simply move x coins around at some pre-agreed time or in a transaction where the data field contains the words "these funds belong to binance". to avoid paying transaction fees, you could sign an off-chain message instead; both bitcoin and ethereum have standards for off-chain signed messages. there are two practical problems with this simple proof-of-assets technique: dealing with cold storage collateral dual-use for safety reasons, most exchanges keep the great majority of customer funds in "cold storage": on offline computers, where transactions need to be signed and carried over onto the internet manually. literal air-gapping is common: a cold storage setup that i used to use for personal funds involved a permanently offline computer generating a qr code containing the signed transaction, which i would scan from my phone. because of the high values at stake, the security protocols used by exchanges are crazier still, and often involve using multi-party computation between several devices to further reduce the chance of a hack against a single device compromising a key. given this kind of setup, making even a single extra message to prove control of an address is an expensive operation! there are several paths that an exchange can take: keep a few public long-term-use addresses. the exchange would generate a few addresses, publish a proof of each address once to prove ownership, and then use those addresses repeatedly. this is by far the simplest option, though it does add some constraints in how to preserve security and privacy. have many addresses, prove a few randomly. the exchange would have many addresses, perhaps even using each address only once and retiring it after a single transaction. in this case, the exchange may have a protocol where from time to time a few addresses get randomly selected and must be "opened" to prove ownership. some exchanges already do something like this with an auditor, but in principle this technique could be turned into a fully automated procedure. more complicated zkp options. for example, an exchange could set all of its addresses to be 1-of-2 multisigs, where one of the keys is different per address, and the other is a blinded version of some "grand" emergency backup key stored in some complicated but very high-security way, eg. a 12-of-16 multisig. to preserve privacy and avoid revealing the entire set of its addresses, the exchange could even run a zero knowledge proof over the blockchain where it proves the total balance of all addresses on chain that have this format. the other major issue is guarding against collateral dual-use. shuttling collateral back and forth between each other to do proof of reserves is something that exchanges could easily do, and would allow them to pretend to be solvent when they actually are not. ideally, proof of solvency would be done in real time, with a proof that updates after every block. if this is impractical, the next best thing would be to coordinate on a fixed schedule between the different exchanges, eg. proving reserves at 1400 utc every tuesday. one final issue is: can you do proof-of-assets on fiat? exchanges don't just hold cryptocurrency, they also hold fiat currency within the banking system. here, the answer is: yes, but such a procedure would inevitably rely on "fiat" trust models: the bank itself can attest to balances, auditors can attest to balance sheets, etc. given that fiat is not cryptographically verifiable, this is the best that can be done within that framework, but it's still worth doing. an alternative approach would be to cleanly separate between one entity that runs the exchange and deals with asset-backed stablecoins like usdc, and another entity (usdc itself) that handles the cash-in and cash-out process for moving between crypto and traditional banking systems. because the "liabilities" of usdc are just on-chain erc20 tokens, proof of liabilities comes "for free" and only proof of assets is required. plasma and validiums: can we make cexes non-custodial? suppose that we want to go further: we don't want to just prove that the exchange has the funds to pay back its users. rather, we want to prevent the exchange from stealing users' funds completely. the first major attempt at this was plasma, a scaling solution that was popular in ethereum research circles in 2017 and 2018. plasma works by splitting up the balance into a set of individual "coins", where each coin is assigned an index and lives in a particular position in the merkle tree of a plasma block. making a valid transfer of a coin requires putting a transaction into the correct position of a tree whose root gets published on-chain. oversimplified diagram of one version of plasma. coins are held in a smart contract that enforces the rules of the plasma protocol at withdrawal time. omisego attempted to make a decentralized exchange based on this protocol, but since then they have pivoted to other ideas as has, for that matter, plasma group itself, which is now the optimistic evm rollup project optimism. it's not worth looking at the technical limitations of plasma as conceived in 2018 (eg. proving coin defragmentation) as some kind of morality tale about the whole concept. since the peak of plasma discourse in 2018, zk-snarks have become much more viable for scaling-related use cases, and as we have said above, zk-snarks change everything. the more modern version of the plasma idea is what starkware calls a validium: basically the same as a zk-rollup, except where data is held off-chain. this construction could be used for a lot of use cases, conceivably anything where a centralized server needs to run some code and prove that it's executing code correctly. in a validium, the operator has no way to steal funds, though depending on the details of the implementation some quantity of user funds could get stuck if the operator disappears. this is all really good: far from cex vs dex being a binary, it turns out that there is a whole spectrum of options, including various forms of hybrid centralization where you gain some benefits like efficiency but still have a lot of cryptographic guardrails preventing the centralized operator from engaging in most forms of abuses. but it's worth getting to the fundamental issue with the right half of this design space: dealing with user errors. by far the most important type of error is: what if a user forgets their password, loses their devices, gets hacked, or otherwise loses access to their account? exchanges can solve this problem: first e-mail recovery, and if even that fails, more complicated forms of recovery through kyc. but to be able to solve such problems, the exchange needs to actually have control over the coins. in order to have the ability to recover user accounts' funds for good reasons, exchanges need to have power that could also be used to steal user accounts' funds for bad reasons. this is an unavoidable tradeoff. the ideal long-term solution is to rely on self-custody, in a future where users have easy access to technologies such as multisig and social recovery wallets to help deal with emergency situations. but in the short term, there are two clear alternatives that have clearly distinct costs and benefits: option exchange-side risk user-side risk custodial exchange (eg. coinbase today) user funds may be lost if there is a problem on the exchange side exchange can help recover account non-custodial exchange (eg. uniswap today) user can withdraw even if exchange acts maliciously user funds may be lost if user screws up another important issue is cross-chain support: exchanges need to support many different chains, and systems like plasma and validiums would need to have code written in different languages to support different platforms, and cannot be implemented at all on others (notably bitcoin) in their current form. in the long-term future, this can hopefully be fixed with technological upgrades and standardization; in the short term, however, it's another argument in favor of custodial exchanges remaining custodial for now. conclusions: the future of better exchanges in the short term, there are two clear "classes" of exchanges: custodial exchanges and non-custodial exchanges. today, the latter category is just dexes such as uniswap, and in the future we may also see cryptographically "constrained" cexes where user funds are held in something like a validium smart contract. we may also see half-custodial exchanges where we trust them with fiat but not cryptocurrency. both types of exchanges will continue to exist, and the easiest backwards-compatible way to improve the safety of custodial exchanges is to add proof of reserve. this consists of a combination of proof of assets and proof of liabilities. there are technical challenges in making good protocols for both, but we can and should go as far as possible to make headway in both, and open-source the software and processes as much as possible so that all exchanges can benefit. in the longer-term future, my hope is that we move closer and closer to all exchanges being non-custodial, at least on the crypto side. wallet recovery would exist, and there may need to be highly centralized recovery options for new users dealing with small amounts, as well as institutions that require such arrangements for legal reasons, but this can be done at the wallet layer rather than within the exchange itself. on the fiat side, movement between the traditional banking system and the crypto ecosystem could be done via cash in / cash out processes native to asset-backed stablecoins such as usdc. however, it will still take a while before we can fully get there. dark mode toggle gitcoin grants round 9: the next phase of growth 2021 apr 02 see all posts special thanks to the gitcoin team for feedback and diagrams. special note: any criticism in these review posts of actions taken by people or organizations, especially using terms like "collusion", "bribe" and "cabal", is only in the spirit of analysis and mechanism design, and should not be taken as (especially moral) criticism of the people and organizations themselves. you're all well-intentioned and wonderful people and i love you. gitcoin grants round 9 has just finished, and as usual the round has been a success. along with 500,000 in matching funds, $1.38 million was donated by over 12,000 contributors to 812 different projects, making this the largest round so far. not only old projects, but also new ones, received a large amount of funding, proving the mechanism's ability to avoid entrenchment and adapt to changing circumstances. the new east asia-specific category in the latest two rounds has also been a success, helping to catapult multiple east asian ethereum projects to the forefront. however, with growing scale, round 9 has also brought out unique and unprecedented challenges. the most important among them is collusion and fraud: in round 9, over 15% of contributions were detected as being probably fraudulent. this was, of course, inevitable and expected from the start; i have actually been surprised at how long it has taken for people to start to make serious attempts to exploit the mechanism. the gitcoin team has responded in force, and has published a blog post detailing their strategies for detecting and responding to adversarial behavior along with a general governance overview. however, it is my opinion that to successfully limit adversarial behavior in the long run more serious reforms, with serious sacrifices, are going to be required. many new, and bigger, funders gitcoin continues to be successful in attracting many matching funders this round. badgerdao, a project that describes itself as a "dao dedicated to building products and infrastructure to bring bitcoin to defi", has donated $300,000 to the matching pool the largest single donation so far. other new funders include uniswap, stakefish, maskbook, fireeyes, polygon, sushiswap and thegraph. as gitcoin grants continues to establish itself as a successful home for ethereum public goods funding, it is also continuing to attract legitimacy as a focal point for donations from projects wishing to support the ecosystem. this is a sign of success, and hopefully it will continue and grow further. the next goal should be to get not just one-time contributions to the matching pool, but long-term commitments to repeated contributions (or even newly launched tokens donating a percentage of their holdings to the matching pool)! churn continues to be healthy one long-time concern with gitcoin grants is the balance between stability and entrenchment: if each project's match award changes too much from round to round, then it's hard for teams to rely on gitcoin grants for funding, and if the match awards change too little, it's hard for new projects to get included. we can measure this! to start off, let's compare the top-10 projects in this round to the top-10 projects in the previous round. in all cases, about half of the top-10 carries over from the previous round and about half is new (the flipside, of course is that half the top-10 drops out). the charts are a slight understatement: the gitcoin grants dev fund and poap appear to have dropped out but actually merely changed categories, so something like 40% churn may be a more accurate number. if you check the results from round 8 against round 7, you also get about 50% churn, and comparing round 7 to round 6 gives similar values. hence, it is looking like the degree of churn is stable. to me, it seems like roughly 40-50% churn is a healthy level, balancing long-time projects' need for stability with the need to avoid new projects getting locked out, but this is of course only my subjective judgement. adversarial behavior the challenging new phenomenon this round was the sheer scale of the adversarial behavior that was attempted. in this round, there were two major issues. first, there were large clusters of contributors discovered that were probably a few individual or small closely coordinated groups with many accounts trying to cheat the mechanism. this was discovered by proprietary analysis algorithms used by the gitcoin team. for this round, the gitcoin team, in consultation with the community, decided to eat the cost of the fraud. each project received the maximum of the match award it would receive if fraudulent transactions were accepted and the match award it would receive if they were not; the difference, about $33,000 in total, was paid out of gitcoin's treasury. for future rounds, however, the team aims to be significantly stricter about security. a diagram from the gitcoin team's post describin their process for finding and dealing with adversarial behavior. in the short term, simply ignoring fraud and accepting its costs has so far worked okay. in the long term, however, fraud must be dealt with, and this raises a challenging political concern. the algorithms that the gitcoin team used to detect the adversarial behavior are proprietary and closed-source, and they have to be closed-source because otherwise the attackers could adapt and get around them. hence, the output of the quadratic funding round is not just decided by a clear mathematical formula of the inputs. rather, if fraudulent transactions were to be removed, it would also be fudged by what risks becoming a closed group twiddling with the outputs according to their arbitrary subjective judgements. it is worth stressing that this is not gitcoin's fault. rather, what is happening is that gitcoin has gotten big enough that it has finally bumped into the exact same problem that every social media site, no matter how well-meaning its team, has been bumping into for the past twenty years. reddit, despite its well-meaning and open-source-oriented team, employs many secretive tricks to detect and clamp down on vote manipulation, as does every other social media site. this is because making algorithms that prevent undesired manipulation, but continue to do so despite the attackers themselves knowing what these algorithms are, is really hard. in fact, the entire science of mechanism design is a half-century-long effort to try to solve this problem. sometimes, there are successes. but often, they keep running into the same challenge: collusion. it turns out that it's not that hard to make mechanisms that give the outcomes you want if all of the participants are acting independently, but once you admit the possibility of one individual controlling many accounts, the problem quickly becomes much harder (or even intractable). but the fact that we can't achieve perfection doesn't mean that we can't try to come closer, and benefit from coming closer. good mechanisms and opaque centralized intervention are substitutes: the better the mechanism, the closer to a good result the mechanism gets all by itself, and the more the secretive moderation cabal can go on vacation (an outcome that the actually-quite-friendly-and-cuddly and decentralization-loving gitcoin moderation cabal very much wants!). in the short term, the gitcoin team is also proactively taking a third approach: making fraud detection and response accountable by inviting third-party analysis and community oversight. picture courtesy of the gitcoin team's excellent blog post. inviting community oversight is an excellent step in preserving the mechanism's legitimacy, and in paving the way for an eventual decentralization of the gitcoin grants institution. however, it's not a 100% solution: as we've seen with technocratic organizations inside national governments, it's actually quite easy for them to retain a large amount of power despite formal democratic oversight and control. the long-term solution is shoring up gitcoin's passive security, so that active security of this type becomes less necessary. one important form of passive security is making some form of unique-human verification no longer optional, but instead mandatory. gitcoin already adds the option to use phone number verification, brightid and several other techniques to "improve an account's trust score" and get greater matching. but what gitcoin will likely be forced to do is make it so that some verification is required to get any matching at all. this will be a reduction in convenience, but the effects can be mitigated by the gitcoin team's work on enabling more diverse and decentralized verification options, and the long-term benefit in enabling security without heavy reliance on centralized moderation, and hence getting longer-lasting legitimacy, is very much worth it. retroactive airdrops a second major issue this round had to do with maskbook. in february, maskbook announced a token and the token distribution included a retroactive airdrop to anyone who had donated to maskbook in previous rounds. the table from maskbook's announcement post showing who is eligible for the airdrops. the controversy was that maskbook was continuing to maintain a gitcoin grant this round, despite now being wealthy and having set a precedent that donors to their grant might be rewarded in the future. the latter issue was particularly problematic as it could be construed as a form of obfuscated vote buying. fortunately, the situation was defused quickly; it turned out that the maskbook team had simply forgotten to consider shutting down the grant after they released their token, and they agreed to shut it down. they are now even part of the funders' league, helping to provide matching funds for future rounds! another project attempted what some construed as a "wink wink nudge nudge" strategy of obfuscated vote buying: they hinted in chat rooms that they have a gitcoin grant and they are going to have a token. no explicit promise to reward contributors was made, but there's a case that the people reading those messages could have interpreted it as such. in both cases, what we are seeing is that collusion is a spectrum, not a binary. in fact, there's a pretty wide part of the spectrum that even completely well-meaning and legitimate projects and their contributors could easily engage in. note that this is a somewhat unusual "moral hierarchy". normally, the more acceptable motivations would be the altruistic ones, and the less acceptable motivations would be the selfish ones. here, though, the motivations closest to the left and the right are selfish; the altruistic motivation is close to the left, but it's not the only motivation close to the left. the key differentiator is something more subtle: are you contributing because you like the consequences of the project getting funded (inside-the-mechanism), or are you contributing because you like some (outside-the-mechanism) consequences of you personally funding the project? the latter motivation is problematic because it subverts the workings of quadratic funding. quadratic funding is all about assuming that people contribute because they like the consequences of the project getting funded, recognizing that the amounts that people contribute will be much less than they ideally "should be" due to the tragedy of the commons, and mathematically compensating for that. but if there are large side-incentives for people to contribute, and these side-incentives are attached to that person specifically and so they are not reduced by the tragedy of the commons at all, then the quadratic matching magnifies those incentives into a very large distortion. in both cases (maskbook, and the other project), we saw something in the middle. the case of the other project is clear: there was an accusation that they made hints at the possibility of formal compensation, though it was not explicitly promised. in the case of maskbook, it seems as though maskbook did nothing wrong: the airdrop was retroactive, and so none of the contributions to maskbook were "tainted" with impute motives. but the problem is more long-term and subtle: if there's a long-term pattern of projects making retroactive airdrops to gitcoin contributors, then users will feel a pressure to contribute primarily not to projects that they think are public goods, but rather to projects that they think are likely to later have tokens. this subverts the dream of using gitcoin quadratic funding to provide alternatives to token issuance as a monetization strategy. the solution: making bribes (and retroactive airdrops) cryptographically impossible the simplest approach would be to delist projects whose behavior comes too close to collusion from gitcoin. in this case, though, this solution cannot work: the problem is not projects doing airdrops while soliciting contributions, the problem is projects doing airdrops after soliciting contributions. while such a project is still soliciting contributions and hence vulnerable to being delisted, there is no indication that they are planning to do an airdrop. more generally, we can see from the examples above that policing motivations is a tough challenge with many gray areas, and is generally not a good fit for the spirit of mechanism design. but if delisting and policing motivations is not the solution, then what is? the solution comes in the form of a technology called maci. maci is a toolkit that allows you to run collusion-resistant applications, which simultaneously guarantee several key properties: correctness: invalid messages do not get processed, and the result that the mechanism outputs actually is the result of processing all valid messages and correctly computing the result. censorship resistance: if someone participates, the mechanism cannot cheat and pretend they did not participate by selectively ignoring their messages. privacy: no one else can see how each individual participated. collusion resistance: a participant cannot prove to others how they participated, even if they wanted to prove this. collusion resistance is the key property: it makes bribes (or retroactive airdrops) impossible, because users would have no way to prove that they actually contributed to someone's grant or voted for someone or performed whatever other action. this is a realization of the secret ballot concept which makes vote buying impractical today, but with cryptography. the technical description of how this works is not that difficult. users participate by signing a message with their private key, encrypting the signed message to a public key published by a central server, and publishing the encrypted signed message to the blockchain. the server downloads the messages from the blockchain, decrypts them, processes them, and outputs the result along with a zk-snark to ensure that they did the computation correctly. users cannot prove how they participated, because they have the ability to send a "key change" message to trick anyone trying to audit them: they can first send a key change message to change their key from a to b, and then send a "fake message" signed with a. the server would reject the message, but no one else would have any way of knowing that the key change message had ever been sent. there is a trust requirement on the server, though only for privacy and coercion resistance; the server cannot publish an incorrect result either by computing incorrectly or by censoring messages. in the long term, multi-party computation can be used to decentralize the server somewhat, strengthening the privacy and coercion resistance guarantees. there is already a quadratic funding system using maci: clr.fund. it works, though at the moment proof generation is still quite expensive; ongoing work on the project will hopefully decrease these costs soon. practical concerns note that adopting maci does come with necessary sacrifices. in particular, there would no longer be the ability to see who contributed to what, weakening gitcoin's "social" aspects. however, the social aspects could be redesigned and changed by taking insights from elections: elections, despite their secret ballot, frequently give out "i voted" stickers. they are not "secure" (in that a non-voter can easily get one), but they still serve the social function. one could go further while still preserving the secret ballot property: one could make a quadratic funding setup where maci outputs the value of how much each participant contributed, but not who they contributed to. this would make it impossible for specific projects to pay people to contribute to them, but would still leave lots of space for users to express their pride in contributing. projects could airdrop to all gitcoin contributors without discriminating by project, and announce that they're doing this together with a link to their gitcoin profile. however, users would still be able to contribute to someone else and collect the airdrop; hence, this would arguably be within bounds of fair play. however, this is still a longer-term concern; maci is likely not ready to be integrated for round 10. for the next few rounds, focusing on stepping up unique-human verification is still the best priority. some ongoing reliance on centralized moderation will be required, though hopefully this can be simultaneously reduced and made more accountable to the community. the gitcoin team has already been taking excellent steps in this direction. and if the gitcoin team does successfully play their role as pioneers in being the first to brave and overcome these challenges, then we will end up with a secure and scalable quadratic funding system that is ready for much broader mainstream applications! light clients and proof of stake | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search light clients and proof of stake posted by vitalik buterin on january 10, 2015 research & development special thanks to vlad zamfir and jae kwon for many of the ideas described in this post aside from the primary debate around weak subjectivity, one of the important secondary arguments raised against proof of stake is the issue that proof of stake algorithms are much harder to make light-client friendly. whereas proof of work algorithms involve the production of block headers which can be quickly verified, allowing a relatively small chain of headers to act as an implicit proof that the network considers a particular history to be valid, proof of stake is harder to fit into such a model. because the validity of a block in proof of stake relies on stakeholder signatures, the validity depends on the ownership distribution of the currency in the particular block that was signed, and so it seems, at least at first glance, that in order to gain any assurances at all about the validity of a block, the entire block must be verified. given the sheer importance of light client protocols, particularly in light of the recent corporate interest in "internet of things" applications (which must often necessarily run on very weak and low-power hardware), light client friendliness is an important feature for a consensus algorithm to have, and so an effective proof of stake system must address it. light clients in proof of work in general, the core motivation behind the "light client" concept is as follows. by themselves, blockchain protocols, with the requirement that every node must process every transaction in order to ensure security, are expensive, and once a protocol gets sufficiently popular the blockchain becomes so big that many users become not even able to bear that cost. the bitcoin blockchain is currently 27 gb in size, and so very few users are willing to continue to run "full nodes" that process every transaction. on smartphones, and especially on embedded hardware, running a full node is outright impossible. hence, there needs to be some way in which a user with far less computing power to still get a secure assurance about various details of the blockchain state what is the balance/state of a particular account, did a particular transaction process, did a particular event happen, etc. ideally, it should be possible for a light client to do this in logarithmic time that is, squaring the number of transactions (eg. going from 1000 tx/day to 1000000 tx/day) should only double a light client's cost. fortunately, as it turns out, it is quite possible to design a cryptocurrency protocol that can be securely evaluated by light clients at this level of efficiency. basic block header model in ethereum (note that ethereum has a merkle tree for transactions and accounts in each block, allowing light clients to easily access more data) in bitcoin, light client security works as follows. instead of constructing a block as a monolithic object containing all of the transactions directly, a bitcoin block is split up into two parts. first, there is a small piece of data called the block header, containing three key pieces of data: the hash of the previous block header the merkle root of the transaction tree (see below) the proof of work nonce additional data like the timestamp is also included in the block header, but this is not relevant here. second, there is the transaction tree. transactions in a bitcoin block are stored in a data structure called a merkle tree. the nodes on the bottom level of the tree are the transactions, and then going up from there every node is the hash of the two nodes below it. for example, if the bottom level had sixteen transactions, then the next level would have eight nodes: hash(tx[1] + tx[2]), hash(tx[3] + tx[4]), etc. the level above that would have four nodes (eg. the first node is equal to hash(hash(tx[1] + tx[2]) + hash(tx[3] + tx[4]))), the level above has two nodes, and then the level at the top has one node, the merkle root of the entire tree. the merkle root can be thought of as a hash of all the transactions together, and has the same properties that you would expect out of a hash if you change even one bit in one transaction, the merkle root will end up completely different, and there is no way to come up with two different sets of transactions that have the same merkle root. the reason why this more complicated tree construction needs to be used is that it actually allows you to come up with a compact proof that one particular transaction was included in a particular block. how? essentially, just provide the branch of the tree going down to the transaction: the verifier will verify only the hashes going down along the branch, and thereby be assured that the given transaction is legitimately a member of the tree that produced a particular merkle root. if an attacker tries to change any hash anywhere going down the branch, the hashes will no longer match and the proof will be invalid. the size of each proof is equal to the depth of the tree ie. logarithmic in the number of transactions. if your block contains 220 (ie. ~1 million) transactions, then the merkle tree will have only 20 levels, and so the verifier will only need to compute 20 hashes in order to verify a proof. if your block contains 230 (ie. ~1 billion) transactions, then the merkle tree will have 30 levels, and so a light client will be able to verify a transaction with just 30 hashes. ethereum extends this basic mechanism with a two additional merkle trees in each block header, allowing nodes to prove not just that a particular transaction occurred, but also that a particular account has a particular balance and state, that a particular event occurred, and even that a particular account does not exist. verifying the roots now, this transaction verification process all assumes one thing: that the merkle root is trusted. if someone proves to you that a transaction is part of a merkle tree that has some root, that by itself means nothing; membership in a merkle tree only proves that a transaction is valid if the merkle root is itself known to be valid. hence, the other critical part of a light client protocol is figuring out exactly how to validate the merkle roots or, more generally, how to validate the block headers. first of all, let us determine exactly what we mean by "validating block headers". light clients are not capable of fully validating a block by themselves; protocols exist for doing validation collaboratively, but this mechanism is expensive, and so in order to prevent attackers from wasting everyone's time by throwing around invalid blocks we need a way of first quickly determining whether or not a particular block header is probably valid. by "probably valid" what we mean is this: if an attacker gives us a block that is determined to be probably valid, but is not actually valid, then the attacker needs to pay a high cost for doing so. even if the attacker succeeds in temporarily fooling a light client or wasting its time, the attacker should still suffer more than the victims of the attack. this is the standard that we will apply to proof of work, and proof of stake, equally. in proof of work, the process is simple. the core idea behind proof of work is that there exists a mathematical function which a block header must satisfy in order to be valid, and it is computationally very intensive to produce such a valid header. if a light client was offline for some period of time, and then comes back online, then it will look for the longest chain of valid block headers, and assume that that chain is the legitimate blockchain. the cost of spoofing this mechanism, providing a chain of block headers that is probably-valid-but-not-actually-valid, is very high; in fact, it is almost exactly the same as the cost of launching a 51% attack on the network. in bitcoin, this proof of work condition is simple: sha256(block_header) < 2**187 (in practice the "target" value changes, but once again we can dispense of this in our simplified analysis). in order to satisfy this condition, miners must repeatedly try different nonce values until they come upon one such that the proof of work condition for the block header is satisfied; on average, this consumes about 269 computational effort per block. the elegant feature of bitcoin-style proof of work is that every block header can be verified by itself, without relying on any external information at all. this means that the process of validating the block headers can in fact be done in constant time download 80 bytes and run a hash of it even better than the logarithmic bound that we have established for ourselves. in proof of stake, unfortunately we do not have such a nice mechanism. light clients in proof of stake if we want to have an effective light client for proof of stake, ideally we would like to achieve the exact same complexity-theoretic properties as proof of work, although necessarily in a different way. once a block header is trusted, the process for accessing any data from the header is the same, so we know that it will take a logarithmic amount of time in order to do. however, we want the process of validating the block headers themselves to be logarithmic as well. to start off, let us describe an older version of slasher, which was not particularly designed to be explicitly light-client friendly: in order to be a "potential blockmaker" or "potential signer", a user must put down a security deposit of some size. this security deposit can be put down at any time, and lasts for a long period of time, say 3 months. during every time slot t (eg. t = 3069120 to 3069135 seconds after genesis), some function produces a random number r (there are many nuances behind making the random number secure, but they are not relevant here). then, suppose that the set of potential signers ps (stored in a separate merkle tree) has size n. we take ps[sha3(r) % n] as the blockmaker, and ps[sha3(r + 1) % n], ps[sha3(r + 2) % n] ... ps[sha3(r + 15) % n] as the signers (essentially, using r as entropy to randomly select a signer and 15 blockmakers) blocks consist of a header containing (i) the hash of the previous block, (ii) the list of signatures from the blockmaker and signers, and (iii) the merkle root of the transactions and state, as well as (iv) auxiliary data like the timestamp. a block produced during time slot t is valid if that block is signed by the blockmaker and at least 10 of the 15 signers. if a blockmaker or signer legitimately participates in the blockmaking process, they get a small signing reward. if a blockmaker or signer signs a block that is not on the main chain, then that signature can be submitted into the main chain as "evidence" that the blockmaker or signer is trying to participate in an attack, and this leads to that blockmaker or signer losing their deposit. the evidence submitter may receive 33% of the deposit as a reward. unlike proof of work, where the incentive not to mine on a fork of the main chain is the opportunity cost of not getting the reward on the main chain, in proof of stake the incentive is that if you mine on the wrong chain you will get explicitly punished for it. this is important; because a very large amount of punishment can be meted out per bad signature, a much smaller number of block headers are required to achieve the same security margin. now, let us examine what a light client needs to do. suppose that the light client was last online n blocks ago, and wants to authenticate the state of the current block. what does the light client need to do? if a light client already knows that a block b[k] is valid, and wants to authenticate the next block b[k+1], the steps are roughly as follows: compute the function that produces the random value r during block b[k+1] (computable either constant or logarithmic time depending on implementation) given r, get the public keys/addresses of the selected blockmaker and signer from the blockchain's state tree (logarithmic time) verify the signatures in the block header against the public keys (constant time) and that's it. now, there is one gotcha. the set of potential signers may end up changing during the block, so it seems as though a light client might need to process the transactions in the block before being able to compute ps[sha3(r + k) % n]. however, we can resolve this by simply saying that it's the potential signer set from the start of the block, or even a block 100 blocks ago, that we are selecting from. now, let us work out the formal security assurances that this protocol gives us. suppose that a light client processes a set of blocks, b[1] ... b[n], such that all blocks starting from b[k + 1] are invalid. assuming that all blocks up to b[k] are valid, and that the signer set for block b[i] is determined from block b[i 100], this means that the light client will be able to correctly deduce the signature validity for blocks b[k + 1] ... b[k + 100]. hence, if an attacker comes up with a set of invalid blocks that fool a light client, the light client can still be sure that the attacker will still have to pay ~1100 security deposits for the first 100 invalid blocks. for future blocks, the attacker will be able to get away with signing blocks with fake addresses, but 1100 security deposits is an assurance enough, particularly since the deposits can be variably sized and thus hold many millions of dollars of capital altogether. thus, even this older version of slasher is, by our definition, light-client-friendly; we can get the same kind of security assurance as proof of work in logarithmic time. a better light-client protocol however, we can do significantly better than the naive algorithm above. the key insight that lets us go further is that of splitting the blockchain up into epochs. here, let us define a more advanced version of slasher, that we will call "epoch slasher". epoch slasher is identical to the above slasher, except for a few other conditions: define a checkpoint as a block such that block.number % n == 0 (ie. every n blocks there is a checkpoint). think of n as being somewhere around a few weeks long; it only needs to be substantially less than the security deposit length. for a checkpoint to be valid, 2/3 of all potential signers have to approve it. also, the checkpoint must directly include the hash of the previous checkpoint. the set of signers during a non-checkpoint block should be determined from the set of signers during the second-last checkpoint. this protocol allows a light client to catch up much faster. instead of processing every block, the light client would skip directly to the next checkpoint, and validate it. the light client can even probabilistically check the signatures, picking out a random 80 signers and requesting signatures for them specifically. if the signatures are invalid, then we can be statistically certain that thousands of security deposits are going to get destroyed. after a light client has authenticated up to the latest checkpoint, the light client can simply grab the latest block and its 100 parents, and use a simpler per-block protocol to validate them as in the original slasher; if those blocks end up being invalid or on the wrong chain, then because the light client has already authenticated the latest checkpoint, and by the rules of the protocol it can be sure that the deposits at that checkpoint are active until at least the next checkpoint, once again the light client can be sure that at least 1100 deposits will be destroyed. with this latter protocol, we can see that not only is proof of stake just as capable of light-client friendliness as proof of work, but moreover it's actually even more light-client friendly. with proof of work, a light client synchronizing with the blockchain must download and process every block header in the chain, a process that is particularly expensive if the blockchain is fast, as is one of our own design objectives. with proof of stake, we can simply skip directly to the latest block, and validate the last 100 blocks before that to get an assurance that if we are on the wrong chain, at least 1100 security deposits will be destroyed. now, there is still a legitimate role for proof of work in proof of stake. in proof of stake, as we have seen, it takes a logarithmic amount of effort to probably-validate each individual block, and so an attacker can still cause light clients a logarithmic amount of annoyance by broadcasting bad blocks. proof of work alone can be effectively validated in constant time, and without fetching any data from the network. hence, it may make sense for a proof of stake algorithm to still require a small amount of proof of work on each block, ensuring that an attacker must spend some computational effort in order to even slightly inconvenience light clients. however, the amount of computational effort required to compute these proofs of work will only need to be miniscule. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle a philosophy of blockchain validation 2020 aug 17 see all posts see also: a proof of stake design philosophy the meaning of decentralization engineering security through coordination problems one of the most powerful properties of a blockchain is the fact that every single part of the blockchain's execution can be independently validated. even if a great majority of a blockchain's miners (or validators in pos) get taken over by an attacker, if that attacker tries to push through invalid blocks, the network will simply reject them. even those users that were not verifying blocks at that time can be (potentially automatically) warned by those who were, at which point they can check that the attacker's chain is invalid, and automatically reject it and coordinate on accepting a chain that follows the rules. but how much validation do we actually need? do we need a hundred independent validating nodes, a thousand? do we need a culture where the average person in the world runs software that checks every transaction? it's these questions that are a challenge, and a very important challenge to resolve especially if we want to build blockchains with consensus mechanisms better than the single-chain "nakamoto" proof of work that the blockchain space originally started with. why validate? a 51% attack pushing through an invalid block. we want the network to reject the chain! there are two main reasons why it's beneficial for a user to validate the chain. first, it maximizes the chance that the node can correctly determine and say on the canonical chain the chain that the community accepts as legitimate. typically, the canonical chain is defined as something like "the valid chain that has the most miners/validators supporting it" (eg. the "longest valid chain" in bitcoin). invalid chains are rejected by definition, and if there is a choice between multiple valid chains, the chain that has the most support from miners/validators wins out. and so if you have a node that verifies all the validity conditions, and hence detects which chains are valid and which chains are not, that maximizes your chances of correctly detecting what the canonical chain actually is. but there is also another deeper reason why validating the chain is beneficial. suppose that a powerful actor tries to push through a change to the protocol (eg. changing the issuance), and has the support of the majority of miners. if no one else validates the chain, this attack can very easily succeed: everyone's clients will, by default, accept the new chain, and by the time anyone sees what is going on, it will be up to the dissenters to try to coordinate a rejection of that chain. but if average users are validating, then the coordination problem falls on the other side: it's now the responsibility of whoever is trying to change the protocol to convince the users to actively download the software patch to accept the protocol change. if enough users are validating, then instead of defaulting to victory, a contentious attempt to force a change of the protocol will default to chaos. defaulting to chaos still causes a lot of disruption, and would require out-of-band social coordination to resolve, but it places a much larger barrier in front of the attacker, and makes attackers much less confident that they will be able to get away with a clean victory, making them much less motivated to even try to start an attack. if most users are validating (directly or indirectly), and an attack has only the support of the majority of miners, then the attack will outright default to failure the best outcome of all. the definition view versus the coordination view note that this reasoning is very different from a different line of reasoning that we often hear: that a chain that changes the rules is somehow "by definition" not the correct chain, and that no matter how many other users accept some new set of rules, what matters is that you personally can stay on the chain with the old rules that you favor. here is one example of the "by definition" perspective from gavin andresen: here's another from the wasabi wallet; this one comes even more directly from the perspective of explaining why full nodes are valuable: notice two core components of this view: a version of the chain that does not accept the rules that you consider fundamental and non-negotiable is by definition not bitcoin (or not ethereum or whatever other chain), not matter how many other people accept that chain. what matters is that you remain on a chain that has rules that you consider acceptable. however, i believe this "individualist" view to be very wrong. to see why, let us take a look at the scenario that we are worried about: the vast majority of participants accept some change to protocol rules that you find unacceptable. for example, imagine a future where transaction fees are very low, and to keep the chain secure, almost everyone else agrees to change to a new set of rules that increases issuance. you stubbornly keep running a node that continues to enforce the old rules, and you fork off to a different chain than the majority. from your point of view, you still have your coins in a system that runs on rules that you accept. but so what? other users will not accept your coins. exchanges will not accept your coins. public websites may show the price of the new coin as being some high value, but they're referring to the coins on the majority chain; your coins are valueless. cryptocurrencies and blockchains are fundamentally social constructs; without other people believing in them, they mean nothing. so what is the alternative view? the core idea is to look at blockchains as engineering security through coordination problems. normally, coordination problems in the world are a bad thing: while it would be better for most people if the english language got rid of its highly complex and irregular spelling system and made a phonetic one, or if the united states switched to metric, or if we could immediately drop all prices and wages by ten percent in the event of a recession, in practice this requires everyone to agree on the switch at the same time, and this is often very very hard. with blockchain applications, however, we are using coordination problems to our advantage. we are using the friction that coordination problems create as a bulwark against malfeasance by centralized actors. we can build systems that have property x, and we can guarantee that they will preserve property x because changing the rules from x to not-x would require a whole bunch of people to agree to update their software at the same time. even if there is an actor that could force the change, doing so would be hard much much harder than it would be if it were instead the responsibility of users to actively coordinate dissent to resist a change. note one particular consequence of this view: it's emphatically not the case that the purpose of your full node is just to protect you, and in the case of a contentious hard fork, people with full nodes are safe and people without full nodes are vulnerable. rather, the perspective here is much more one of herd immunity: the more people are validating, the more safe everyone is, and even if only some portion of people are validating, everyone gets a high level of protection as a result. looking deeper into validation we now get to the next topic, and one that is very relevant to topics such as light clients and sharding: what are we actually accomplishing by validating? to understand this, let us go back to an earlier point. if an attack happens, i would argue that we have the following preference order over how the attack goes: default to failure > default to chaos > default to victory the ">" here of course means "better than". the best is if an attack outright fails; the second best is if an attack leads to confusion, with everyone disagreeing on what the correct chain is, and the worst is if an attack succeeds. why is chaos so much better than victory? this is a matter of incentives: chaos raises costs for the attacker, and denies them the certainty that they will even win, discouraging attacks from being attempted in the first place. a default-to-chaos environment means that an attacker needs to win both the blockchain war of making a 51% attack and the "social war" of convincing the community to follow along. this is much more difficult, and much less attractive, than just launching a 51% attack and claiming victory right there. the goal of validation is then to move away from default to victory to (ideally) default to failure or (less ideally) default to chaos. if you have a fully validating node, and an attacker tries to push through a chain with different rules, then the attack fails. if some people have a fully validating node but many others don't, the attack leads to chaos. but now we can think: are there other ways of achieving the same effect? light clients and fraud proofs one natural advancement in this regard is light clients with fraud proofs. most blockchain light clients that exist today work by simply validating that the majority of miners support a particular block, and not bothering to check if the other protocol rules are being enforced. the client runs on the trust assumption that the majority of miners is honest. if a contentious fork happens, the client follows the majority chain by default, and it's up to users to take an active step if they want to follow the minority chain with the old rules; hence, today's light clients under attack default to victory. but with fraud proof technology, the situation starts to look very different. a fraud proof in its simplest form works as follows. typically, a single block in a blockchain only touches a small portion of the blockchain "state" (account balances, smart contract code....). if a fully verifying node processes a block and finds that it is invalid, they can generate a package (the fraud proof) containing the block along with just enough data from the blockchain state to process the block. they broadcast this package to light clients. light clients can then take the package and use that data to verify the block themselves, even if they have no other data from the chain. a single block in a blockchain touches only a few accounts. a fraud proof would contain the data in those accounts along with merkle proofs proving that that data is correct. this technique is also sometimes known as stateless validation: instead of keeping a full database of the blockchain state, clients can keep only the block headers, and they can verify any block in real time by asking other nodes for a merkle proof for any desired state entries that block validation is accessing. the power of this technique is that light clients can verify individual blocks only if they hear an alarm (and alarms are verifiable, so if a light client hears a false alarm, they can just stop listening to alarms from that node). hence, under normal circumstances, the light client is still light, checking only which blocks are supported by the majority of miners/validators. but under those exceptional circumstances where the majority chain contains a block that the light client would not accept, as long as there is at least one honest node verifying the fraudulent block, that node will see that it is invalid, broadcast a fraud proof, and thereby cause the rest of the network to reject it. sharding sharding is a natural extension of this: in a sharded system, there are too many transactions in the system for most people to be verifying directly all the time, but if the system is well designed then any individual invalid block can be detected and its invalidity proven with a fraud proof, and that proof can be broadcasted across the entire network. a sharded network can be summarized as everyone being a light client. and as long as each shard has some minimum threshold number of participants, the network has herd immunity. in addition, the fact that in a sharded system block production (and not just block verification) is highly accessible and can be done even on consumer laptops is also very important. the lack of dependence on high-performance hardware at the core of the network ensures that there is a low bar on dissenting minority chains being viable, making it even harder for a majority-driven protocol change to "win by default" and bully everyone else into submission. this is what auditability usually means in the real world: not that everyone is verifying everything all the time, but that (i) there are enough eyes on each specific piece that if there is an error it will get detected, and (ii) when an error is detected that fact that be made clear and visible to all. that said, in the long run blockchains can certainly improve on this. one particular source of improvements is zk-snarks (or "validity proofs"): efficiently verifiably cryptographic proofs that allow block producers to prove to clients that blocks satisfy some arbitrarily complex validity conditions. validity proofs are stronger than fraud proofs because they do not depend on an interactive game to catch fraud. another important technology is data availability checks, which can protect against blocks whose data is not fully published. data availability checks do rely on a very conservative assumption that there exists at least some small number of honest nodes somewhere in the network continues to apply, though the good news is that this minimum honesty threshold is low, and does not grow even if there is a very large number of attackers. timing and 51% attacks now, let us get to the most powerful consequence of the "default to chaos" mindset: 51% attacks themselves. the current norm in many communities is that if a 51% attack wins, then that 51% attack is necessarily the valid chain. this norm is often stuck to quite strictly; and a recent 51% attack on ethereum classic illustrated this quite well. the attacker reverted more than 3000 blocks (stealing 807,260 etc in a double-spend in the process), which pushed the chain farther back in history than one of the two etc clients (openethereum) was technically able to revert; as a result, geth nodes went with the attacker's chain but openethereum nodes stuck with the original chain. we can say that the attack did in fact default to chaos, though this was an accident and not a deliberate design decision of the etc community. unfortunately, the community then elected to accept the (longer) attack chain as canonical, a move described by the eth_classic twitter as "following proof of work as intended". hence, the community norms actively helped the attacker win. but we could instead agree on a definition of the canonical chain that works differently: particularly, imagine a rule that once a client has accepted a block as part of the canonical chain, and that block has more than 100 descendants, the client will from then on never accept a chain that does not include that block. alternatively, in a finality-bearing proof of stake setup (which eg. ethereum 2.0 is), imagine a rule that once a block is finalized it can never be reverted. 5 block revert limit only for illustration purposes; in reality the limit could be something longer like 100-1000 blocks. to be clear, this introduces a significant change to how canonicalness is determined: instead of clients just looking at the data they receive by itself, clients also look at when that data was received. this introduces the possibility that, because of network latency, clients disagree: what if, because of a massive attack, two conflicting blocks a and b finalize at the same time, and some clients see a first and some see b first? but i would argue that this is good: it means that instead of defaulting to victory, even 51% attacks that just try to revert transactions default to chaos, and out-of-band emergency response can choose which of the two blocks the chain continues with. if the protocol is well-designed, forcing an escalation to out-of-band emergency response should be very expensive: in proof of stake, such a thing would require 1/3 of validators sacrificing their deposits and getting slashed. potentially, we could expand this approach. we could try to make 51% attacks that censor transactions default to chaos too. research on timeliness detectors pushes things further in the direction of attacks of all types defaulting to failure, though a little chaos remains because timeliness detectors cannot help nodes that are not well-connected and online. for a blockchain community that values immutability, implementing revert limits of this kind are arguably the superior path to take. it is difficult to honestly claim that a blockchain is immutable when no matter how long a transaction has been accepted in a chain, there is always the possibility that some unexpected activity by powerful actors can come along and revert it. of course, i would claim that even btc and etc do already have a revert limit at the extremes; if there was an attack that reverted weeks of activity, the community would likely adopt a user-activated soft fork to reject the attackers' chain. but more definitively agreeing on and formalizing this seems like a step forward. conclusion there are a few "morals of the story" here. first, if we accept the legitimacy of social coordination, and we accept the legitimacy of indirect validation involving "1-of-n" trust models (that is, assuming that there exists one honest person in the network somewhere; not the same as assuming that one specific party, eg. infura, is honest), then we can create blockchains that are much more scalable. second, client-side validation is extremely important for all of this to work. a network where only a few people run nodes and everyone else really does trust them is a network that can easily be taken over by special interests. but avoiding such a fate does not require going to the opposite extreme and having everyone always validate everything! systems that allow each individual block to be verified in isolation, so users only validate blocks if someone else raises an alarm, are totally reasonable and serve the same effect. but this requires accepting the "coordination view" of what validation is for. third, if we allow the definition of canonicalness includes timing, then we open many doors in improving our ability to reject 51% attacks. the easiest property to gain is weak subjectivity: the idea that if clients are required to log on at least once every eg. 3 months, and refuse to revert longer than that, then we can add slashing to proof of stake and make attacks very expensive. but we can go further: we can reject chains that revert finalized blocks and thereby protect immutability, and even protect against censorship. because the network is unpredictable, relying on timing does imply attacks "defaulting to chaos" in some cases, but the benefits are very much worth it. with all of these ideas in mind, we can avoid the traps of (i) over-centralization, (ii) overly redundant verification leading to inefficiency and (iii) misguided norms accidentally making attacks easier, and better work toward building more resilient, performant and secure blockchains. mihai’s ethereum project update. the first year. | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search mihai’s ethereum project update. the first year. posted by mihai alisie on march 14, 2015 research & development into the ether hello, fellow ethereans. i am mihai alisie, one of the ethereum founders, and for the past year i’ve served as the vice president of the ethereum foundation and director of ethereum switzerland. in the early days of the project i concentrated my attention on the community side and worked on the holons idea for a distributed physical infrastructure; however, the project needed someone in switzerland, and i was more useful here focusing on other things. during the the last couple of weeks i started to think about how far we’ve come and how many things we have accomplished since this project started. from idea to implementation and everything in between, it was (and continues to be) the most intense ride of my life. while contemplating the things that have happened during the past year, i felt inspired to write this post. as i started to go through the posts, tweets and messages published over the last months, the article started to take the shape of a timeline that offers a window into the heart of this project. i hope you will enjoy it. looking back. what started as a vision. logic will get you from a to b. imagination will take you everywhere. albert einstein i vividly recall the autumn of 2013 when i first encountered vitalik’s ethereum idea concentrated in the initial lines of what was going to become the ethereum whitepaper. he sent it to me in an email, mentioning that the message had been sent only to a select few. ethereum felt different from anything else he’d sent me before. the same rush i felt when i first understood the implications of bitcoin was returning, this time with thousands of new multi-dimensional rabbit holes to explore. myself and vitalik go way back to 2011, when we first got in touch and started bitcoin magazine together. a few months after exchanging our first messages over the internet, we published the first issue and had a team of passionate people making it happen with nothing more than skype, google docs and little to no money. it definitely wasn’t easy, but it was fun. the mag pushed us into growing at an accelerated rate, each in his own way, opening in the process many opportunities for the both of us. along the way i have gained a deep respect for vitalik as i’ve watched him grow as a colleague, mind and person. this is why i felt both honored and incredibly excited when he invited me to be one of the founders of this project in late 2013. and then the rollercoaster left the station. between december 2013 and january 2014 the focus was mainly on how to bootstrap the vision set forth by vitalik in the ethereum whitepaper. we all ended up agreeing that having a genesis sale was an exciting idea; however, it was also one of the most challenging at a strategic level. after long discussions on multiple levels, we decided to postpone the genesis sale from happening in february in order to create a proper infrastructure and legal strategy since we were navigating in deep, uncharted territories. with that in mind, we scouted a number of jurisdictions, hoping to find one that would work as a base for the project. it was certainly a discouraging effort at times, considering the lack of legal clarity associated with the space we operate in, but we kept pushing. this is how and why, at the end of january 2014, i was landing in zurich with the mission of scouting switzerland as a potential jurisdiction for what became in the meantime the ethereum project. switzerland looked interesting from the beginning thanks to its strategic positioning in the heart of europe, being at the same time a financial nexus with a friendly business environment, the world’s center for human rights and a global freedom of speech and innovative technology hub. we were not the only ones to see these things as great, in 12 may silent circle joining us in the beautiful switzerland for similar reasons. the stage has been set. we couch-surfed for a while, moving six times in five weeks through various airbnbs while keeping focused on the mission, and in a short time the trip to switzerland turned to be an oasis of hope for ethereum. thanks to the friendly and warm swiss crypto community we were not completely alone and we received invaluable local input with regards to what to do and where. in less than a month since we first touched down at the zurich airport we had a swiss entity up and running and a legal strategy starting to take shape. a big “big up” goes especially to the guys from open transactions and bitcoin suisse for their help and for the strategic swiss networks they shared with us. february 2014 was a wild month, with progress being made on all fronts: community growth, codebase, wiki content, business infrastructure and legal strategy. during this month vitalik revealed ethereum for the first time at the miami conference, we had our first reddit ama, and the core development team solidified into a world-class crypto crew. after the miami conference, gavin wood and jeffrey wilcke came on board full-time, although they were working on ethereum out of pure passion months before this while building the official c++ and go clients. since our launch strategy had changed, instead of launching the genesis sale immediately we chose to release the open source alpha testnet client first. it turned out to be an excellent decision as it brought us increasing attention from the crypto community as a whole, since people were able to see and test that the thing actually works. march began in force with the proof of concept 3 client being released on the first. on march 5th, after a long search, we finally moved into our ethereum spaceship located in the beautiful canton of zug.   the "spaceship” marks the birth of holon 000, our swiss ethereum hq. between march and may, an area that received considerable attention was the regulatory side of the project. even if it was frustrating at times, during these months we worked hard on crafting the legal strategy and the optimal organizational structure for the ethereum project while surfing the murky worldwide regulatory waters. after months of strategic legal grinding, on may 20th the digital finance compliance association was created with the purpose of interfacing with the swiss authorities and providing guidance while also providing assistance and advice for current and new members. dfca will most likely become more prominent in the coming months as it will enlarge its focus and interact with other worldwide jurisdictions. in the meantime, for those doing crypto projects in switzerland or looking to relocate into a friendlier jurisdiction, you can get in touch with dfca for help and/or more info. june brought a nice surprise in the form of the thiel fellowship being awarded to vitalik for the cutting-edge work he has been doing with ethereum and elsewhere. at the same time it felt like an official recognition of the efforts put into creating this new decentralized ecosystem. on the other ethereum project fronts, by june we advanced to the fourth version of the proof of concept implementations, rapidly moving towards the fifth version; our twitter followers number surpassed 6,000; and the number of global meetups that focused on ethereum reached 58 groups in 49 different cities, spread over 19 countries. things started to get big. internally, during this period we had intense soul-searching sessions while also engaging in conversations with the community regarding the nature of the project. all the discussions converged into one major decision: should we go for-profit or non-profit? the decision was to pursue the non-profit route in order to make sure that the vision does not get corrupted. we all saw what “not being evil” means for some people when profit is involved. the community applauded the decision. in july the idea of a 12 sec block-time was taking shape, we finally had our worldwide legal strategy set up, and the final genesis sale preparations were in full swing – with a strong focus on the design, creation and testing of the sale app and multi-sig system. in a true “victory comes to those who are prepared” fashion, taylor and marian managed to deliver a system able to support a flux of over 31’500 btc flawlessly.  july 14th marked the birth of the swiss ethereum foundation, followed on the 22nd by the release of the proof of concept 5 and culminating in a bang on the 24th with the start of the genesis sale synchronized with a second reddit ama. i have to admit that we all had high hopes, but no one was anticipating that in 24 hours we would surpass any previous initiative in the space. in any case, it was one of the most fulfilling birthday presents ever and proof that we weren’t crazy – or that there are many other crazy people out there and we’d found each other ^_^ in the first part of august the sale price of ether started to increase from 2000 eth/btc to 19xx eth/btc towards the last rate of 1337 eth/btc while steady progress was being made on the technical front. vitalik compared the state of ethereum at that date with the state of bitcoin in mid-2009. some catch-up was in order. as the genesis sale was approaching its end, the focus of the project shifted towards building the ethereum core and scaling up the operations. this is also why we took the decision to start ramping up development and global operations before the genesis sale ended. it turned out to be quite a challenge to establish the banking and financial infrastructure required for a project of this scale; however, after many searches and discussions with different entities, we found the solutions we needed. in the end, according to swiss law, through the genesis sale we bootstrapped a peer-to-peer platform on which decentralized applications can be developed, deployed and utilized. ether, the world's first and only cryptofuel, serves as a critical component that powers all computational steps and storage operations on the system. without this mechanism, ethereum would be subject to denial of service attacks, spam and would quickly grind to a halt as malicious or errant developers triggered infinite computational loops and other bad things. it was not an easy thing to explain but luckily for us there are open minds out there. after 42 days the genesis sale ended, and after pinching each other to see if we were dreaming, we realized that we’d touched a historical milestone. it’s still hard to believe, but we pulled off the biggest crowdfunding-like “one-time” initiative in the history of the internet. ever. according to wikipedia, ethereum is rated as the second-biggest crowdfunded project in the history of the internet – sitting proudly next to the first occupant that raised over $70 mil, but over the course of years, not 42 days. if you take a look at the list i think one of the most important things to observe is the nature of the project and the amount of support put into it. sure, we’ve seen gadgets, consoles and video games raising millions in the past – but this time it’s about something serious and game-changing. more precisely, a swiss-based non-profit, bootstrapped via crypto creating an open source, blockchain-based platform for building the next internet and gifting it to the world. right after this monumental achievement, things got even bigger. paul brody (ibm) was tweeting openly about ibm forking ethereum and playing with it for their vision of the internet of things. it was certainly an exciting announcement and made some people look at ethereum with different eyes . moving forward, the autumn of 2014 was a dev team tour de force, both in terms of coding and in operations. by october the berlin hub was operational and hacking hard, with the operations ramping up at an incredible rate, with october 5th marking the release of our poc 6. it was a big release for a number of reasons, one of the highlights being the blockchain speed. the block time was decreased from 60 seconds to 12 seconds, using a new ghost-based protocol that expanded upon our previous efforts at reducing the block time to 60 seconds. but that was not all: before the month ended, on the 26th we touched another milestone – the first whisper-based dapp was live. on november 3rd we hit another "world's first" milestone – the world's first ethereum halloween costume, created by a community member :) meanwhile, by november 7th the amsterdam hub was also operational and ready to “go go go”. the month of november peaked with vitalik edging out mark zuckerberg to receive the world technology awards prize for the most innovative software in 2014 and continuing with the first ethereum developer conference devcon 0 and a parallel hackathon on transparency organized in the canton of geneva. in the finals of the world technology event, vitalik faced four other nominees: jack/laura dangermond, christian lanng, kira radinsky and (drums in the background) mark zuckerberg. however, ethereum is innovation on a whole different level and apparently we are not the only ones thinking so. in a way, this event felt similar to the thiel fellowship: it was yet another official acknowledgement regarding the innovative potential this technology has. aside from this, i think it's also proof that there are many entrepreneurs and innovators out there keeping a close eye on #ethereum. the hackathon on transparency was held on november 26th and was organized by professor jean henry morin and stephan tual at cui geneva, with the main topic being "rebooting public administration & democracy". a diverse group of about 50 people attended the event and we split the presentation in two rooms – one for less-technical people and one for those that brought their laptops looking to build things on ethereum. after the presentation i gave with taylor ended, we mixed the participants and started the brainstorming session, with everyone thinking about cool ways in which blockchains can be used for transparency in public administration. all in all it was a great success, the night ending with professor morin attending a roundtable discussion with the canton of geneva on transparency and the advantages blockchain technologies like ethereum bring on this front. ethereum devcon was tremendously productive, and necessary, as it allowed us to have nearly all of the developers together and really get to know each other, disseminate a lot of information and chat over a lot of ideas. five days of back-to-back presentations, coding and ethereum discussions synced the distributed team members on the same wavelength. you can read a really nice recap of the conference here and watch the presentations given here. before november ended, we also launched a community survey and the results were fantastic. it felt good to know that the community is with us, appreciating the work put into making this happen. after all, the project would be nothing without an awesome community to take things forward after the launch and we're all aware of it. december brought new team members, the release of the proof of concept 7 and new projects such as the network intelligence dashboard. on the december 8th the first whisper was sent between go & c++ implementations. no one in between heard a thing :) in the background, the security audits were about to get started, so the ethereum stress-testing team was formed. together with the security auditors, they help us reveal weaknesses inside the system for fixing. to date, no show-stopping flaws have been found and the audits are progressing nicely. in january 2015 we launched the poc 7 and v met with the guys from ibm and samsung at ces where they were showing what they’ve built with ethereum. it was both amazing and encouraging to see a few proof of concepts built in such a short time using ethereum, even if at the time it was in a crude state.   january also brought the launch of our bug bounty program, looking to encourage people from all around the world to poke at our code and expose the things that might have escaped our eyes and minds. in sum,  january was a month of constant progress and optimizations with a few events spicing things up here and there. february brought the release of the poc 8 release, the first whisper group chat inside mist (our ethereum browser), and an increasing attention on the genesis launch and the future plans at project level. ethereum was also recently asked to join the world wide web consortium (w3c) and participate in the web payments interest group. the membership includes some of the largest banks, payments companies, tech companies and telecoms in the world. ethereum had a strong presence at the most recent meeting of the group (feb 2–4) in utrecht and is likely to have significant input into crafting the designs proposed, use cases built and standards forged by this committee. the ultimate goal of the web payments group is to deliver the payments component of the html6 spec. the road toward the ultimate release of that spec and incorporation by the browser manufacturers will be paved by the design, execution and refinement of many use cases, and ethereum will both lead that development effort and serve as the leading candidate technology for much of it. below you can see the group picture taken at the last w3c organized in utrecht, with joseph lubin and vinay gupta sitting on the right of the table. march started with a series of announcements regarding the genesis launch and intense work sessions of proof of concept 9. vinay gupta wrote a comprehensive article where he explained the codenames and what they mean for both users and developers, while gavin has started documenting the development for the multi-phase ethereum launch. all in all, it looks like an extremely busy and exciting period is coming for ethereum. so…here we are, today, one year after we settled in our swiss hq – days away from launching the ethereum network. we’ve gone from a few guys trying to figure out how to bootstrap this project to a worldwide movement of thousands of people, over 100 meetups in 33 countries, hundreds of amazing volunteers and a world-class core team of over 40 members. we’re now a planetary community of people learning, sharing and co-creating at light-speed. epic. and things haven't even started yet. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle should there be demand-based recurring fees on ens domains? 2022 sep 09 see all posts special thanks to lars doucet, glen weyl and nick johnson for discussion and feedback on various topics. ens domains today are cheap. very cheap. the cost to register and maintain a five-letter domain name is only $5 per year. this sounds reasonable from the perspective of one person trying to register a single domain, but it looks very different when you look at the situation globally: when ens was younger, someone could have registered all 8938 five-letter words in the scrabble wordlist (which includes exotic stuff like "burrs", "fluyt" and "zoril") and pre-paid their ownership for a hundred years, all for the price of a dozen lambos. and in fact, many people did: today, almost all of those five-letter words are already taken, many by squatters waiting for someone to buy the domain from them at a much higher price. a random scrape of opensea shows that about 40% of all these domains are for sale or have been sold on that platform alone. the question worth asking is: is this really the best way to allocate domains? by selling off these domains so cheaply, ens dao is almost certainly gathering far less revenue than it could, which limits its ability to act to improve the ecosystem. the status quo is also bad for fairness: being able to buy up all the domains cheaply was great for people in 2017, is okay in 2022, but the consequences may severely handicap the system in 2050. and given that buying a five-letter-word domain in practice costs anywhere from 0.1 to 500 eth, the notionally cheap registration prices are not actually providing cost savings to users. in fact, there are deep economic reasons to believe that reliance on secondary markets makes domains more expensive than a well-designed in-protocol mechanism. could we allocate ongoing ownership of domains in a better way? is there a way to raise more revenue for ens dao, do a better job of ensuring domains go to those who can make best use of them, and at the same time preserve the credible neutrality and the accessible very strong guarantees of long-term ownership that make ens valuable? problem 1: there is a fundamental tradeoff between strength of property rights and fairness suppose that there are \(n\) "high-value names" (eg. five-letter words in the scrabble dictionary, but could be any similar category). suppose that each year, users grab up \(k\) names, and some portion \(p\) of them get grabbed by someone who's irrationally stubborn and not willing to give them up (\(p\) could be really low, it just needs to be greater than zero). then, after \(\frac{n}{k * p}\) years, no one will be able to get a high-value name again. this is a two-line mathematical theorem, and it feels too simple to be saying anything important. but it actually gets at a crucial truth: time-unlimited allocation of a finite resource is incompatible with fairness across long time horizons. this is true for land; it's the reason why there have been so many land reforms throughout history, and it's a big part of why many advocate for land taxes today. it's also true for domains, though the problem in the traditional domain space has been temporarily alleviated by a "forced dilution" of early .com holders in the form of a mass introduction of .io, .me, .network and many other domains. ens has soft-committed to not add new tlds to avoid polluting the global namespace and rupturing its chances of eventual integration with mainstream dns, so such a dilution is not an option. fortunately, ens charges not just a one-time fee to register a domain, but also a recurring annual fee to maintain it. not all decentralized domain name systems had the foresight to implement this; unstoppable domains did not, and even goes so far as to proudly advertise its preference for short-term consumer appeal over long-term sustainability ("no renewal fees ever!"). the recurring fees in ens and traditional dns are a healthy mitigation to the worst excesses of a truly unlimited pay-once-own-forever model: at the very least, the recurring fees mean that no one will be able to accidentally lock down a domain forever through forgetfulness or carelessness. but it may not be enough. it's still possible to spend $500 to lock down an ens domain for an entire century, and there are certainly some types of domains that are in high enough demand that this is vastly underpriced. problem 2: speculators do not actually create efficient markets once we admit that a first-come-first-serve model with low fixed fees has these problems, a common counterargument is to say: yes, many of the names will get bought up by speculators, but speculation is natural and good. it is a free market mechanism, where speculators who actually want to maximize their profit are motivated to resell the domain in such a way that it goes to whoever can make the best use of the domain, and their outsized returns are just compensation for this service. but as it turns out, there has been academic research on this topic, and it is not actually true that profit-maximizing auctioneers maximize social welfare! quoting myerson 1981: by announcing a reservation price of 50, the seller risks a probability \((1 / 2^n)\) of keeping the object even though some bidder is willing to pay more than \(t_0\) for it; but the seller also increases his expected revenue, because he can command a higher price when the object is sold. thus the optimal auction may not be ex-post efficient. to see more clearly why this can happen, consider the example in the above paragraph, for the case when \(n = 1\) ... ex post efficiency would require that the bidder must always get the object, as long as his value estimate is positive. but then the bidder would never admit to more than an infinitesimal value estimate, since any positive bid would win the object ... in fact the seller's optimal policy is to refuse to sell the object for less than 50. translated into diagram form: maximizing revenue for the seller almost always requires accepting some probability of never selling the domain at all, leaving it unused outright. one important nuance in the argument is that seller-revenue-maximizing auctions are at their most inefficient when there is one possible buyer (or at least, one buyer with a valuation far above the others), and the inefficiency decreases quickly once there are many competing potential buyers. but for a large class of domains, the first category is precisely the situation they are in. domains that are simply some person, project or company's name, for example, have one natural buyer: that person or project. and so if a speculator buys up such a name, they will of course set the price high, accepting a large chance of never coming to a deal to maximize their revenue in the case where a deal does arise. hence, we cannot say that speculators grabbing a large portion of domain allocation revenues is merely just compensation for them ensuring that the market is efficient. on the contrary, speculators can easily make the market worse than a well-designed mechanism in the protocol that encourages domains to be directly available for sale at fair prices. one cheer for stricter property rights: stability of domain ownership has positive externalities the monopoly problems of overly-strict property rights on non-fungible assets have been known for a long time. resolving this issue in a market-based way was the original goal of harberger taxes: require the owner of each covered asset to set a price at which they are willing to sell it to anyone else, and charge an annual fee based on that price. for example, one could charge 0.5% of the sale price every year. holders would be incentivized to leave the asset available for purchase at prices that are reasonable, "lazy" holders who refuse to sell would lose money every year, and hoarding assets without using them would in many cases become economically infeasible outright. but the risk of being forced to sell something at any time can have large economic and psychological costs, and it's for this reason that advocates of harberger taxes generally focus on industrial property applications where the market participants are sophisticated. where do domains fall on the spectrum? let us consider the costs of a business getting "relocated", in three separate cases: a data center, a restaurant, and an ens name. data center restaurant ens name confusion from people expecting old location an employee comes to the old location, and unexpectedly finds it closed. an employee or a customer comes to the old location, and unexpectedly finds it closed. someone sends a big chunk of money to the wrong address. loss of location-specific long-term investment low the restaurant will probably lose many long-term customers for whom the new location is too far away the owner spent years building a brand around the old name that cannot easily carry over. as it turns out, domains do not hold up very well. domain name owners are often not sophisticated, the costs of switching domain names are often high, and negative externalities of a name-change gone wrong can be large. the highest-value owner of coinbase.eth may not be coinbase; it could just as easily be a scammer who would grab up the domain and then immediately make a fake charity or ico claiming it's run by coinbase and ask people to send that address their money. for these reasons, harberger taxing domains is not a great idea. alternative solution 1: demand-based recurring pricing maintaining ownership over an ens domain today requires paying a recurring fee. for most domains, this is a simple and very low $5 per year. the only exceptions are four-letter domains ($160 per year) and three-letter domains ($640 per year). but what if instead, we make the fee somehow depend on the actual level of market demand for the domain? this would not be a harberger-like scheme where you have to make the domain available for immediate sale at a particular price. rather, the initiative in the price-setting procedure would fall on the bidders. anyone could bid on a particular domain, and if they keep an open bid for a sufficiently long period of time (eg. 4 weeks), the domain's valuation rises to that level. the annual fee on the domain would be proportional to the valuation (eg. it might be set to 0.5% of the valuation). if there are no bids, the fee might decay at a constant rate. when a bidder sends their bid amount into a smart contract to place a bid, the owner has two options: they could either accept the bid, or they could reject, though they may have to start paying a higher price. if a bidder bids a value higher than the actual value of the domain, the owner could sell to them, costing the bidder a huge amount of money. this property is important, because it means that "griefing" domain holders is risky and expensive, and may even end up benefiting the victim. if you own a domain, and a powerful actor wants to harass or censor you, they could try to make a very high bid for that domain to greatly increase your annual fee. but if they do this, you could simply sell to them and collect the massive payout. this already provides much more stability and is more noob-friendly than a harberger tax. domain owners don't need to constantly worry whether or not they're setting prices too low. rather, they can simply sit back and pay the annual fee, and if someone offers to bid they can take 4 weeks to make a decision and either sell the domain or continue holding it and accept the higher fee. but even this probably does not provide quite enough stability. to go even further, we need a compromise on the compromise. alternative solution 2: capped demand-based recurring pricing we can modify the above scheme to offer even stronger guarantees to domain-name holders. specifically, we can try to offer the following property: strong time-bound ownership guarantee: for any fixed number of years, it's always possible to compute a fixed amount of money that you can pre-pay to unconditionally guarantee ownership for at least that number of years. in math language, there must be some function \(y = f(n)\) such that if you pay \(y\) dollars (or eth), you get a hard guarantee that you will be able to hold on to the domain for at least \(n\) years, no matter what happens. \(f\) may also depend on other factors, such as what happened to the domain previously, as long as those factors are known at the time the transaction to register or extend a domain is made. note that the maximum annual fee after \(n\) years would be the derivative \(f'(n)\). the new price after a bid would be capped at the implied maximum annual fee. for example, if \(f(n) = \frac{1}{2}n^2\), so \(f'(n) = n\), and you get a bid of $5 after 7 years, the annual fee would rise to $5, but if you get a bid of $10 after 7 years, the annual fee would only rise to $7. if no bids that raise the fee to the max are made for some length of time (eg. a full year), \(n\) resets. if a bid is made and rejected, \(n\) resets. and of course, we have a highly subjective criterion that \(f(n)\) must be "reasonable". we can create compromise proposals by trying different shapes for \(f\): type \(f(n)\) (\(p_0\) = price of last sale or last rejected bid, or $1 if most recent event is a reset) in plain english total cost to guarantee holding for >= 10 years total cost to guarantee holding for >= 100 years exponential fee growth \(f(n) = \int_0^n p_0 * 1.1^n\) the fee can grow by a maximum of 10% per year (with compounding). $836 $7.22m linear fee growth \(f(n) = p_0 * n + \frac{15}{2}n^2\) the annual fee can grow by a maximum of $15 per year. $1250 $80k capped annual fee \(f(n) = 640 * n\) the annual fee cannot exceed $640 per year. that is, a domain in high demand can start to cost as much as a three-letter domain, but not more. $6400 $64k or in chart form: note that the amounts in the table are only the theoretical maximums needed to guarantee holding a domain for that number of years. in practice, almost no domains would have bidders willing to bid very high amounts, and so holders of almost all domains would end up paying much less than the maximum. one fascinating property of the "capped annual fee" approach is that there are versions of it that are strictly more favorable to existing domain-name holders than the status quo. in particular, we could imagine a system where a domain that gets no bids does not have to pay any annual fee, and a bid could increase the annual fee to a maximum of $5 per year. demand from external bids clearly provides some signal about how valuable a domain is (and therefore, to what extent an owner is excluding others by maintaining control over it). hence, regardless of your views on what level of fees should be required to maintain a domain, i would argue that you should find some parameter choice for demand-based fees appealing. i will still make my case for why some superlinear \(f(n)\), a max annual fee that goes up over time, is a good idea. first, paying more for longer-term security is a common feature throughout the economy. fixed-rate mortgages usually have higher interest rates than variable-rate mortgages. you can get higher interest by providing deposits that are locked up for longer periods of time; this is compensation the bank pays you for providing longer-term security to the bank. similarly, longer-term government bonds typically have higher yields. second, the annual fee should be able to eventually adjust to whatever the market value of the domain is; we just don't want that to happen too quickly. superlinear \(f(n)\) values still make hard guarantees of ownership reasonably accessible over pretty long timescales: with the linear-fee-growth formula \(f(n) = p_0 * n + \frac{15}{2}n^2\), for only $6000 ($120 per year) you could ensure ownership of the domain for 25 years, and you would almost certainly pay much less. the ideal of "register and forget" for censorship-resistant services would still be very much available. from here to there weakening property norms, and increasing fees, is psychologically very unappealing to many people. this is true even when these fees make clear economic sense, and even when you can redirect fee revenue into a ubi and mathematically show that the majority of people would economically net-benefit from your proposal. cities have a hard time adding congestion pricing, even when it's painfully clear that the only two choices are paying congestion fees in dollars and paying congestion fees in wasted time and weakened mental health driving in painfully slow traffic. land value taxes, despite being in many ways one of the most effective and least harmful taxes out there, have a hard time getting adopted. unstoppable domains's loud and proud advertisement of "no renewal fees ever" is in my view very short-sighted, but it's clearly at least somewhat effective. so how could i possibly think that we have any chance of adding fees and conditions to domain name ownership? the crypto space is not going to solve deep challenges in human political psychology that humanity has failed at for centuries. but we do not have to. i see two possible answers that do have some realistic hope for success: democratic legitimacy: come up with a compromise proposal that really is a sufficient compromise that it makes enough people happy, and perhaps even makes some existing domain name holders (not just potential domain name holders) better off than they are today. for example, we could implement demand-based annual fees (eg. setting the annual fee to 0.5% of the highest bid) with a fee cap of $640 per year for domains up to eight letters long, and $5 per year for longer domains, and let domain holders pay nothing if no one makes a bid. many average users would save money under such a proposal. market legitimacy: avoid the need to get legitimacy to overturn people's expectations in the existing system by instead creating a new system (or sub-system). in traditional dns, this could be done just by creating a new tld that would be as convenient as existing tlds. in ens, there is a stated desire to stick to .eth only to avoid conflicting with the existing domain name system. and using existing subdomains doesn't quite work: foo.bar.eth is much less nice than foo.eth. one possible middle route is for the ens dao to hand off single-letter domain names solely to projects that run some other kind of credibly-neutral marketplace for their subdomains, as long as they hand over at least 50% of the revenue to the ens dao. for example, perhaps x.eth could use one of my proposed pricing schemes for its subdomains, and t.eth could implement a mechanism where ens dao has the right to forcibly transfer subdomains for anti-fraud and trademark reasons. foo.x.eth just barely looks good enough to be sort-of a substitute for foo.eth; it will have to do. if making changes to ens domain pricing itself are off the table, then the market-based approach of explicitly encouraging marketplaces with different rules in subdomains should be strongly considered. to me, the crypto space is not just about coins, and i admit my attraction to ens does not center around some notion of unconditional and infinitely strict property-like ownership over domains. rather, my interest in the space lies more in credible neutrality, and property rights that are strongly protected particularly against politically motivated censorship and arbitrary and targeted interference by powerful actors. that said, a high degree of guarantee of ownership is nevertheless very important for a domain name system to be able to function. the hybrid proposals i suggest above are my attempt at preserving total credible neutrality, continuing to provide a high degree of ownership guarantee, but at the same time increasing the cost of domain squatting, raising more revenue for the ens dao to be able to work on important public goods, and improving the chances that people who do not have the domain they want already will be able to get one. dark mode toggle updating my blog: a quick gpt chatbot coding experiment 2022 dec 06 see all posts the gpt chatbot has been all the rage the last few days. along with many important use cases like writing song lyrics, acting as a language learning buddy and coming up with convincing-sounding arguments for arbitrary political opinions, one of the things that many people are excited about is the possibility of using the chatbot to write code. in a lot of cases, it can succeed and write some pretty good code especially for common tasks. in cases that cover less well-trodden ground, however, it can fail: witness its hilariously broken attempt to write a plonk verifier: (in case you want to know how to do it kinda-properly, here is a plonk verifier written by me) but how well do these tools actually perform in the average case? i decided to take the gpt3 chatbot for a spin, and see if i could get it to solve a problem very relevant to me personally: changing the ipfs hash registered in my vitalik.eth ens record, in order to make the new article that i just released on my blog viewable through ens. the process of updating the ens view of my blog normally consists of two steps: first, publish the updated contents to ipfs, and second, update my ens record to contain the ipfs hash of the new contents. fleek has automated the first part of this for me for a long time: i just push the contents to github, and fleek uploads the new version to ipfs automatically. i have been told that i could change the settings to give fleek the power to also edit my ens, but here i want to be fully "self-sovereign" and not trust third parties, so i have not done this. instead, so far, i have had to go to the gui at app.ens.domains, click a few times, wait for a few loading screens to pass, and finally click "add / edit record", change the content hash and click "confirm". this is all a cumbersome process, and so today i finally thought that i would write a script in javascript to automate this all down to a single piece of javascript that i could just copy-paste into my browser console in the future. the task is simple: send an ethereum transaction to the right address with the right calldata to update the content hash record in the ens contract to equal the ipfs hash that fleek gives to me. yesterday, i did this all manually (twice, once to publish and again to add some corrections), and the ipfs hashes i got were: bafybeifvvseiarzdfoqadphxtfu5yjfgj3cr6x344qce4s4f7wqyf3zv4e bafybeieg6fhbjlhkzhbyfnmyid3ko5ogxp3mykdarsfyw66lmq6lq5z73m if you click through to the top article in each one, you'll see the two different versions. this hash format is often called a "bafyhash", because the hashes all begin with "bafy". but there is a problem: the format of the hash that is saved in ethereum is not a bafyhash. here's the calldata of the transaction that made one of the update operations: yes, i checked, that is not hexadecimalized ascii. i do know that the ipfs content hash is the last two rows of the data. how do i know? well, i checked the two different transactions i sent for my two updates, and i saw that the top row is the same and the bottom two rows are different. good enough. so what do i do to convert from a bafyhash into a binary ipfs content hash? well, let me try asking the gpt3 chatbot! noooo!!!!!!!!!! many issues. first, two things that are my fault: i forgot to mention this, but i wanted javascript, not python. it uses external dependencies. i want my javascript copy-pasteable into the console, so i don't want any libraries. these are on me to specify, though, and in my next instruction to the chatbot i will. but now we get to the things that are its fault: bafyhashes are base 32, not base 58. there is a base-58 format for ipfs hashes, but those are called "qm hashes", not "bafyhashes". by "binary" i didn't want literal ones and zeroes, i wanted the normal binary format, a bytes or bytearray. that said, at this part of the story i did not even realize that bafyhashes are base 32. i fixed the two issues that were my fault first: baaaaaaaaaaaaad, the ai trainer said sheepishly! the atob function is for base 64, not base 58. ok, let's keep going. a few rounds later... it's hard to see what's going on at first, but it's incredibly wrong. basically, instead of converting the whole string from base 58 to base 16, it's converting each individual digit to base 16. not what i want to do! guess i'll have to tell it what strategy it should use: better! i soon start to realize that i don't need base 58, i need base 32, and furthermore i need the lowercase version of base 32. i also want the code wrapped in a function. for these simpler steps, it gets much more cooperative: at this point, i try actually passing the bafyhashes i have into this function, and i get unrecognizably different outputs. looks like i can't just assume this is generic base 32, and i have to poke into the details. hmm, can i perhaps ask the gpt3 chatbot? ok, this is not helpful. let me try to be more specific. this is an.... interesting guess, but it's totally wrong. after this point, i give up on the gpt3 for a while, and keep poking at the generated hex and the actual hex in python until i find similarities. eventually, i figure it out: i actually do convert both hexes to literal binary, and search from a binary substring of one in the other. i discover that there is an offset of 2 bits. i just edit the code manually, compensating for the offset by dividing the bigint by 4: because i already know what to do, i also just code the part that generates the entire calldata myself: anyway, then i switch to the next task: the portion of the javascript that actually sends a transaction. i go back to the gpt3. nooooo! i said no libraries!!!!1!1! i tell it what to use directly: this is more successful. two errors though: a from address actually is required. you can't stick an integer into the gas field, you need a hex value. also, post eip-1559, there really isn't much point in hard-coding a gasprice. from here, i do the rest of the work myself. function bafytohex(bafystring) { // create a lookup table for the base32 alphabet var alphabet = 'abcdefghijklmnopqrstuvwxyz234567'; var base = alphabet.length; var lookuptable = {}; alphabet.split('').foreach(function(char, i) { lookuptable[char] = i; }); // decode the base32-encoded string into a big integer var bigint = bafystring.split('').reduce(function(acc, curr) { return acc * bigint(base) + bigint(lookuptable[curr]); }, bigint(0)) / bigint(4); // convert the big integer into a hexadecimal string var hexstring = bigint.tostring(16); return 'e30101701220' + hexstring.slice(-64); } function bafytocalldata(bafystring) { return ( '0x304e6ade' + 'ee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835' + '0000000000000000000000000000000000000000000000000000000000000040' + '0000000000000000000000000000000000000000000000000000000000000026' + bafytohex(bafystring) + '0000000000000000000000000000000000000000000000000000' ) } async function setbafyhash(bafystring) { calldata = bafytocalldata(bafystring); const addr = (await window.ethereum.enable())[0]; // set the "to" address for the transaction const to = '0x4976fb03c32e5b8cfe2b6ccb31c09ba78ebaba41'; // set the transaction options const options = { from: addr, to: to, data: calldata, gas: "0x040000" }; console.log(options); // send the transaction window.ethereum.send('eth_sendtransaction', [options], function(error, result) { if (error) { console.error(error); } else { console.log(result); } }); } i ask the gpt-3 some minor questions: how to declare an async function, and what keyword to use in twitter search to search only tweets that contain images (needed to write this post). it answers both flawlessly: do async function functionname to declare an async function, and use filter:images to filter for tweets that contain images. conclusions the gpt-3 chatbot was helpful as a programming aid, but it also made plenty of mistakes. ultimately, i was able to get past its mistakes quickly because i had lots of domain knowledge: i know that it was unlikely that browsers would have a builtin for base 58, which is a relatively niche format mostly used in the crypto world, and so i immediately got suspicious of its attempt to suggest atob i could eventually recall that the hash being all-lowercase means it's base 32 and not base 58 i knew that the data in the ethereum transaction had to encode the ipfs hash in some sensible way, which led me to eventually come up with the idea of checking bit offsets i know that a simple "correct" way to convert between base a and base b is to go through some abstract integer representation as an in-between, and that javascript supported big integers. i knew about window.ethereum.send when i got the error that i was not allowed to put an integer into the gas field, i knew immediately that it was supposed to be hex. at this point, ai is quite far from being a substitute for human programmers. in this particular case, it only sped me up by a little bit: i could have figured things out with google eventually, and indeed in one or two places i did go back to googling. that said, it did introduce me to some coding patterns i had not seen before, and it wrote the base converter faster than i would have on my own. for the boilerplate operation of writing the javascript to send a simple transaction, it did quite well. that said, ai is improving quickly and i expect it to keep improving further and ironing out bugs like this over time. addendum: while writing the part of this post that involved more copy-paste than thinking, i put on my music playlist on shuffle. the first song that started playing was, coincidentally, basshunter's boten anna ("anna the bot"). dark mode toggle an incomplete guide to stealth addresses 2023 jan 20 see all posts special thanks to ben difrancesco, matt solomon, toni wahrstätter and antonio sanso for feedback and review one of the largest remaining challenges in the ethereum ecosystem is privacy. by default, anything that goes onto a public blockchain is public. increasingly, this means not just money and financial transactions, but also ens names, poaps, nfts, soulbound tokens, and much more. in practice, using the entire suite of ethereum applications involves making a significant portion of your life public for anyone to see and analyze. improving this state of affairs is an important problem, and this is widely recognized. so far, however, discussions on improving privacy have largely centered around one specific use case: privacy-preserving transfers (and usually self-transfers) of eth and mainstream erc20 tokens. this post will describe the mechanics and use cases of a different category of tool that could improve the state of privacy on ethereum in a number of other contexts: stealth addresses. what is a stealth address system? suppose that alice wants to send bob an asset. this could be some quantity of cryptocurrency (eg. 1 eth, 500 rai), or it could be an nft. when bob receives the asset, he does not want the entire world to know that it was he who got it. hiding the fact that a transfer happened is impossible, especially if it's an nft of which there is only one copy on-chain, but hiding who is the recipient may be much more viable. alice and bob are also lazy: they want a system where the payment workflow is exactly the same as it is today. bob sends alice (or registers on ens) some kind of "address" encoding how someone can pay him, and that information alone is enough for alice (or anyone else) to send him the asset. note that this is a different kind of privacy than what is provided by eg. tornado cash. tornado cash can hide transfers of mainstream fungible assets such as eth or major erc20s (though it's most easily useful for privately sending to yourself), but it's very weak at adding privacy to transfers of obscure erc20s, and it cannot add privacy to nft transfers at all. the ordinary workflow of making a payment with cryptocurrency. we want to add privacy (no one can tell that it was bob who received the asset), but keep the workflow the same. stealth addresses provide such a scheme. a stealth address is an address that can be generated by either alice or bob, but which can only be controlled by bob. bob generates and keeps secret a spending key, and uses this key to generate a stealth meta-address. he passes this meta-address to alice (or registers it on ens). alice can perform a computation on this meta-address to generate a stealth address belonging to bob. she can then send any assets she wants to send to this address, and bob will have full control over them. along with the transfer, she publishes some extra cryptographic data (an ephemeral pubkey) on-chain that helps bob discover that this address belongs to him. another way to look at it is: stealth addresses give the same privacy properties as bob generating a fresh address for each transaction, but without requiring any interaction from bob. the full workflow of a stealth address scheme can be viewed as follows: bob generates his root spending key (m) and stealth meta-address (m). bob adds an ens record to register m as the stealth meta-address for bob.eth. we assume alice knows that bob is bob.eth. alice looks up his stealth meta-address m on ens. alice generates an ephemeral key that only she knows, and that she only uses once (to generate this specific stealth address). alice uses an algorithm that combines her ephemeral key and bob's meta-address to generate a stealth address. she can now send assets to this address. alice also generates her ephemeral public key, and publishes it to the ephemeral public key registry (this can be done in the same transaction as the first transaction sending assets to this stealth address). for bob to discover stealth addresses belonging to him, bob needs to scan the ephemeral public key registry for the entire list of ephemeral public keys published by anyone for any reason since the last time he did the scan. for each ephemeral public key, bob attempts to combine it with his root spending key to generate a stealth address, and checks if there are any assets in that address. if there are, bob computes the spending key for that address and remembers it. this all relies on two uses of cryptographic trickery. first, we need a pair of algorithms to generate a shared secret: one algorithm which uses alice's secret thing (her ephemeral key) and bob's public thing (his meta-address), and another algorithm which uses bob's secret thing (his root spending key) and alice's public thing (her ephemeral public key). this can be done in many ways; diffie-hellman key exchange was one of the results that founded the field of modern cryptography, and it accomplishes exactly this. but a shared secret by itself is not enough: if we just generate a private key from the shared secret, then alice and bob could both spend from this address. we could leave it at that, leaving it up to bob to move the funds to a new address, but this is inefficient and needlessly reduces security. so we also add a key blinding mechanism: a pair of algorithms where bob can combine the shared secret with his root spending key, and alice can combine the shared secret with bob's meta-address, in such a way that alice can generate the stealth address, and bob can generate the spending key for that stealth address, all without creating a public link between the stealth address and bob's meta-address (or between one stealth address and another). stealth addresses with elliptic curve cryptography stealth addresses using elliptic curve cryptography were originally introduced in the context of bitcoin by peter todd in 2014. this technique works as follows (this assumes prior knowledge of basic elliptic curve cryptography; see here, here and here for some tutorials): bob generates a key m, and computes m = g * m, where g is a commonly-agreed generator point for the elliptic curve. the stealth meta-address is an encoding of m. alice generates an ephemeral key r, and publishes the ephemeral public key r = g * r. alice can compute a shared secret s = m * r, and bob can compute the same shared secret s = m * r. in general, in both bitcoin and ethereum (including correctly-designed erc-4337 accounts), an address is a hash containing the public key used to verify transactions from that address. so you can compute the address if you compute the public key. to compute the public key, alice or bob can compute p = m + g * hash(s) to compute the private key for that address, bob (and bob alone) can compute p = m + hash(s) this satisfies all of our requirements above, and is remarkably simple! there is even an eip trying to define a stealth address standard for ethereum today, that both supports this approach and gives space for users to develop other approaches (eg. that support bob having separate spending and viewing keys, or that use different cryptography for quantum-resistant security). now you might think: stealth addresses are not too difficult, the theory is already solid, and getting them adopted is just an implementation detail. the problem is, however, that there are some pretty big implementation details that a truly effective implementation would need to get through. stealth addresses and paying transaction fees suppose that someone sends you an nft. mindful of your privacy, they send it to a stealth address that you control. after scanning the ephem pubkeys on-chain, your wallet automatically discovers this address. you can now freely prove ownership of the nft or transfer it to someone else. but there's a problem! that account has 0 eth in it, and so there is no way to pay transaction fees. even erc-4337 token paymasters won't work, because those only work for fungible erc20 tokens. and you can't send eth into it from your main wallet, because then you're creating a publicly visible link. inserting memes of 2017-era (or older) crypto scams is an important technique that writers can use to signal erudition and respectableness, because it shows that they have been around for a long time and have refined tastes, and are not easily swayed by current-thing scam figures like sbf. there is one "easy" way to solve the problem: just use zk-snarks to transfer funds to pay for the fees! but this costs a lot of gas, an extra hundreds of thousands of gas just for a single transfer. another clever approach involves trusting specialized transaction aggregators ("searchers" in mev lingo). these aggregators would allow users to pay once to purchase a set of "tickets" that can be used to pay for transactions on-chain. when a user needs to spend an nft in a stealth address that contains nothing else, they provide the aggregator with one of the tickets, encoded using a chaumian blinding scheme. this is the original protocol that was used in centralized privacy-preserving e-cash schemes that were proposed in the 1980s and 1990s. the searcher accepts the ticket, and repeatedly includes the transaction in their bundle for free until the transaction is successfully accepted in a block. because the quantity of funds involved is low, and it can only be used to pay for transaction fees, trust and regulatory issues are much lower than a "full" implementation of this kind of centralized privacy-preserving e-cash. stealth addresses and separating spending and viewing keys suppose that instead of bob just having a single master "root spending key" that can do everything, bob wants to have a separate root spending key and viewing key. the viewing key can see all of bob's stealth addresses, but cannot spend from them. in the elliptic curve world, this can be solved using a very simple cryptographic trick: bob's meta-address m is now of the form (k, v), encoding g * k and g * v, where k is the spending key and v is the viewing key. the shared secret is now s = v * r = v * r, where r is still alice's ephemeral key and r is still the ephemeral public key that alice publishes. the stealth address's public key is p = k + g * hash(s) and the private key is p = k + hash(s). notice that the first clever cryptographic step (generating the shared secret) uses the viewing key, and the second clever cryptographic step (alice and bob's parallel algorithms to generate the stealth address and its private key) uses the root spending key. this has many use cases. for example, if bob wants to receive poaps, then bob could give his poap wallet (or even a not-very-secure web interface) his viewing key to scan the chain and see all of his poaps, without giving this interface the power to spend those poaps. stealth addresses and easier scanning to make it easier to scan the total set of ephemeral public keys, one technique is to add a view tag to each ephemeral public key. one way to do this in the above mechanism is to make the view tag be one byte of the shared secret (eg. the x-coordinate of s modulo 256, or the first byte of hash(s)). this way, bob only needs to do a single elliptic curve multiplication for each ephemeral public key to compute the shared secret, and only 1/256 of the time would bob need to do more complex calculation to generate and check the full address. stealth addresses and quantum-resistant security the above scheme depends on elliptic curves, which are great but are unfortunately vulnerable to quantum computers. if quantum computers become an issue, we would need to switch to quantum-resistant algorithms. there are two natural candidates for this: elliptic curve isogenies and lattices. elliptic curve isogenies are a very different mathematical construction based on elliptic curves, that has the linearity properties that let us do similar cryptographic tricks to what we did above, but cleverly avoids constructing cyclic groups that might be vulnerable to discrete logarithm attacks with quantum computers. the main weakness of isogeny-based cryptography is its highly complicated underlying mathematics, and the risk that possible attacks are hidden under this complexity. some isogeny-based protocols were broken last year, though others remain safe. the main strength of isogenies is the relatively small key sizes, and the ability to port over many kinds of elliptic curve-based approaches directly. a 3-isogeny in csidh, source here. lattices are a very different cryptographic construction that relies on far simpler mathematics than elliptic curve isogenies, and is capable of some very powerful things (eg. fully homomorphic encryption). stealth address schemes could be built on lattices, though designing the best one is an open problem. however, lattice-based constructions tend to have much larger key sizes. fully homomorphic encryption, an application of lattices. fhe could also be used to help stealth address protocols in a different way: to help bob outsource the computation of checking the entire chain for stealth addresses containing assets without revealing his view key. a third approach is to construct a stealth address scheme from generic black-box primitives: basic ingredients that lots of people need for other reasons. the shared secret generation part of the scheme maps directly to key exchange, a, errr... important component in public key encryption systems. the harder part is the parallel algorithms that let alice generate only the stealth address (and not the spending key) and let bob generate the spending key. unfortunately, you cannot build stealth addresses out of ingredients that are simpler than what is required to build a public-key encryption system. there is a simple proof of this: you can build a public-key encryption system out of a stealth address scheme. if alice wants to encrypt a message to bob, she can send n transactions, each transaction going to either a stealth address belonging to bob or to a stealth address belonging to herself, and bob can see which transactions he received to read the message. this is important because there are mathematical proofs that you can't do public key encryption with just hashes, whereas you can do zero-knowledge proofs with just hashes hence, stealth addresses cannot be done with just hashes. here is one approach that does use relatively simple ingredients: zero knowledge proofs, which can be made out of hashes, and (key-hiding) public key encryption. bob's meta-address is a public encryption key plus a hash h = hash(x), and his spending key is the corresponding decryption key plus x. to create a stealth address, alice generates a value c, and publishes as her ephemeral pubkey an encryption of c readable by bob. the address itself is an erc-4337 account whose code verifies transactions by requiring them to come with a zero-knowledge proof proving ownership of values x and c such that k = hash(hash(x), c) (where k is part of the account's code). knowing x and c, bob can reconstruct the address and its code himself. the encryption of c tells no one other than bob anything, and k is a hash, so it reveals almost nothing about c. the wallet code itself only contains k, and c being private means that k cannot be traced back to h. however, this requires a stark, and starks are big. ultimately, i think it is likely that a post-quantum ethereum world will involve many applications using many starks, and so i would advocate for an aggregation protocol like that described here to combine all of these starks into a single recursive stark to save space. stealth addresses and social recovery and multi-l2 wallets i have for a long time been a fan of social recovery wallets: wallets that have a multisig mechanism with keys shared between some combination of institutions, your other devices and your friends, where some supermajority of those keys can recover access to your account if you lose your primary key. however, social recovery wallets do not mix nicely with stealth addresses: if you have to recover your account (meaning, change which private key controls it), you would also have to perform some step to change the account verification logic of your n stealth wallets, and this would require n transactions, at a high cost to fees, convenience and privacy. a similar concern exists with the interaction of social recovery and a world of multiple layer-2 protocols: if you have an account on optimism, and on arbitrum, and on starknet, and on scroll, and on polygon, and possibly some of these rollups have a dozen parallel instances for scaling reasons and you have an account on each of those, then changing keys may be a really complex operation. changing the keys to many accounts across many chains is a huge effort. one approach is to bite the bullet and accept that recoveries are rare and it's okay for them to be costly and painful. perhaps you might have some automated software transfer the assets out into new stealth addresses at random intervals over a two-week time span to reduce the effectiveness of time-based linking. but this is far from perfect. another approach is to secret-share the root key between the guardians instead of using smart contract recovery. however, this removes the ability to de-activate a guardian's power to help recover your account, and so is long-run risky. a more sophisticated approach involves zero-knowledge proofs. consider the zkp-based scheme above, but modifying the logic as follows. instead of the account holding k = hash(hash(x), c) directly, the account would hold a (hiding) commitment to the location of k on the chain. spending from that account would then require providing a zero-knowledge proof that (i) you know the location on the chain that matches that commitment, and (ii) the object in that location contains some value k (which you're not revealing), and that you have some values x and c that satisfy k = hash(hash(x), c). this allows many accounts, even across many layer-2 protocols, to be controlled by a single k value somewhere (on the base chain or on some layer-2), where changing that one value is enough to change the ownership of all your accounts, all without revealing the link between your accounts and each other. conclusions basic stealth addresses can be implemented fairly quickly today, and could be a significant boost to practical user privacy on ethereum. they do require some work on the wallet side to support them. that said, it is my view that wallets should start moving toward a more natively multi-address model (eg. creating a new address for each application you interact with could be one option) for other privacy-related reasons as well. however, stealth addresses do introduce some longer-term usability concerns, such as difficulty of social recovery. it is probably okay to simply accept these concerns for now, eg. by accepting that social recovery will involve either a loss of privacy or a two-week delay to slowly release the recovery transactions to the various assets (which could be handled by a third-party service). in the longer term, these problems can be solved, but the stealth address ecosystem of the long term is looking like one that would really heavily depend on zero-knowledge proofs. dark mode toggle coordination, good and bad 2020 sep 11 see all posts special thanks to karl floersch and jinglan wang for feedback and review see also: on collusion engineering security through coordination problems trust models the meaning of decentralization coordination, the ability for large groups of actors to work together for their common interest, is one of the most powerful forces in the universe. it is the difference between a king comfortably ruling a country as an oppressive dictatorship, and the people coming together and overthrowing him. it is the difference between the global temperature going up 3-5'c and the temperature going up by a much smaller amount if we work together to stop it. and it is the factor that makes companies, countries and any social organization larger than a few people possible at all. coordination can be improved in many ways: faster spread of information, better norms that identify what behaviors are classified as cheating along with more effective punishments, stronger and more powerful organizations, tools like smart contracts that allow interactions with reduced levels of trust, governance technologies (voting, shares, decision markets...), and much more. and indeed, we as a species are getting better at all of these things with each passing decade. but there is also a very philosophically counterintuitive dark side to coordination. while it is emphatically true that "everyone coordinating with everyone" leads to much better outcomes than "every man for himself", what that does not imply is that each individual step toward more coordination is necessarily beneficial. if coordination is improved in an unbalanced way, the results can easily be harmful. we can think about this visually as a map, though in reality the map has many billions of "dimensions" rather than two: the bottom-left corner, "every man for himself", is where we don't want to be. the top-right corner, total coordination, is ideal, but likely unachievable. but the landscape in the middle is far from an even slope up, with many reasonably safe and productive places that it might be best to settle down in and many deep dark caves to avoid. now what are these dangerous forms of partial coordination, where someone coordinating with some fellow humans but not others leads to a deep dark hole? it's best to describe them by giving examples: citizens of a nation valiantly sacrificing themselves for the greater good of their country in a war.... when that country turns out to be ww2-era germany or japan a lobbyist giving a politician a bribe in exchange for that politician adopting the lobbyist's preferred policies someone selling their vote in an election all sellers of a product in a market colluding to raise their prices at the same time large miners of a blockchain colluding to launch a 51% attack in all of the above cases, we see a group of people coming together and cooperating with each other, but to the great detriment of some group that is outside the circle of coordination, and thus to the net detriment of the world as a whole. in the first case, it's all the people that were the victims of the aforementioned nations' aggression that are outside the circle of coordination and suffer heavily as a result; in the second and third cases, it's the people affected by the decisions that the corrupted voter and politician are making, in the fourth case it's the customers, and in the fifth case it's the non-participating miners and the blockchain's users. it's not an individual defecting against the group, it's a group defecting against a broader group, often the world as a whole. this type of partial coordination is often called "collusion", but it's important to note that the range of behaviors that we are talking about is quite broad. in normal speech, the word "collusion" tends to be used more often to describe relatively symmetrical relationships, but in the above cases there are plenty of examples with a strong asymmetric character. even extortionate relationships ("vote for my preferred policies or i'll publicly reveal your affair") are a form of collusion in this sense. in the rest of this post, we'll use "collusion" to refer to "undesired coordination" generally. evaluate intentions, not actions (!!) one important property of especially the milder cases of collusion is that one cannot determine whether or not an action is part of an undesired collusion just by looking at the action itself. the reason is that the actions that a person takes are a combination of that person's internal knowledge, goals and preferences together with externally imposed incentives on that person, and so the actions that people take when colluding, versus the actions that people take on their own volition (or coordinating in benign ways) often overlap. for example, consider the case of collusion between sellers (a type of antitrust violation). if operating independently, each of three sellers might set a price for some product between $5 and $10; the differences within the range reflect difficult-to-see factors such as the seller's internal costs, their own willingness to work at different wages, supply-chain issues and the like. but if the sellers collude, they might set a price between $8 and $13. once again, the range reflects different possibilities regarding internal costs and other difficult-to-see factors. if you see someone selling that product for $8.75, are they doing something wrong? without knowing whether or not they coordinated with other sellers, you can't tell! making a law that says that selling that product for more than $8 would be a bad idea; maybe there are legitimate reasons why prices have to be high at the current time. but making a law against collusion, and successfully enforcing it, gives the ideal outcome you get the $8.75 price if the price has to be that high to cover sellers' costs, but you don't get that price if the factors driving prices up naturally are low. this applies in the bribery and vote selling cases too: it may well be the case that some people vote for the orange party legitimately, but others vote for the orange party because they were paid to. from the point of view of someone determining the rules for the voting mechanism, they don't know ahead of time whether the orange party is good or bad. but what they do know is that a vote where people vote based on their honest internal feelings works reasonably well, but a vote where voters can freely buy and sell their votes works terribly. this is because vote selling has a tragedy-of-the-commons: each voter only gains a small portion of the benefit from voting correctly, but would gain the full bribe if they vote the way the briber wants, and so the required bribe to lure each individual voter is far smaller than the bribe that would actually compensate the population for the costs of whatever policy the briber wants. hence, votes where vote selling is permitted quickly collapse into plutocracy. understanding the game theory we can zoom further out and look at this from the perspective of game theory. in the version of game theory that focuses on individual choice that is, the version that assumes that each participant makes decisions independently and that does not allow for the possibility of groups of agents working as one for their mutual benefit, there are mathematical proofs that at least one stable nash equilibrium must exist in any game. in fact, mechanism designers have a very wide latitude to "engineer" games to achieve specific outcomes. but in the version of game theory that allows for the possibility of coalitions working together (ie. "colluding"), called cooperative game theory, we can prove that there are large classes of games that do not have any stable outcome (called a "core"). in such games, whatever the current state of affairs is, there is always some coalition that can profitably deviate from it. one important part of that set of inherently unstable games is majority games. a majority game is formally described as a game of agents where any subset of more than half of them can capture a fixed reward and split it among themselves a setup eerily similar to many situations in corporate governance, politics and many other situations in human life. that is to say, if there is a situation with some fixed pool of resources and some currently established mechanism for distributing those resources, and it's unavoidably possible for 51% of the participants can conspire to seize control of the resources, no matter what the current configuration is there is always some conspiracy that can emerge that would be profitable for the participants. however, that conspiracy would then in turn be vulnerable to potential new conspiracies, possibly including a combination of previous conspirators and victims... and so on and so forth. round a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 this fact, the instability of majority games under cooperative game theory, is arguably highly underrated as a simplified general mathematical model of why there may well be no "end of history" in politics and no system that proves fully satisfactory; i personally believe it's much more useful than the more famous arrow's theorem, for example. note once again that the core dichotomy here is not "individual versus group"; for a mechanism designer, "individual versus group" is surprisingly easy to handle. it's "group versus broader group" that presents the challenge. decentralization as anti-collusion but there is another, brighter and more actionable, conclusion from this line of thinking: if we want to create mechanisms that are stable, then we know that one important ingredient in doing so is finding ways to make it more difficult for collusions, especially large-scale collusions, to happen and to maintain themselves. in the case of voting, we have the secret ballot a mechanism that ensures that voters have no way to prove to third parties how they voted, even if they want to prove it (maci is one project trying to use cryptography to extend secret-ballot principles to an online context). this disrupts trust between voters and bribers, heavily restricting undesired collusions that can happen. in that case of antitrust and other corporate malfeasance, we often rely on whistleblowers and even give them rewards, explicitly incentivizing participants in a harmful collusion to defect. and in the case of public infrastructure more broadly, we have that oh-so-important concept: decentralization. one naive view of why decentralization is valuable is that it's about reducing risk from single points of technical failure. in traditional "enterprise" distributed systems, this is often actually true, but in many other cases we know that this is not sufficient to explain what's going on. it's instructive here to look at blockchains. a large mining pool publicly showing how they have internally distributed their nodes and network dependencies doesn't do much to calm community members scared of mining centralization. and pictures like these, showing 90% of bitcoin hashpower at the time being capable of showing up to the same conference panel, do quite a bit to scare people: but why is this image scary? from a "decentralization as fault tolerance" view, large miners being able to talk to each other causes no harm. but if we look at "decentralization" as being the presence of barriers to harmful collusion, then the picture becomes quite scary, because it shows that those barriers are not nearly as strong as we thought. now, in reality, the barriers are still far from zero; the fact that those miners can easily perform technical coordination and likely are all in the same wechat groups does not, in fact, mean that bitcoin is "in practice little better than a centralized company". so what are the remaining barriers to collusion? some major ones include: moral barriers. in liars and outliers, bruce schneier reminds us that many "security systems" (locks on doors, warning signs reminding people of punishments...) also serve a moral function, reminding potential misbehavers that they are about to conduct a serious transgression and if they want to be a good person they should not do that. decentralization arguably serves that function. internal negotiation failure. the individual companies may start demanding concessions in exchange for participating in the collusion, and this could lead to negotiation stalling outright (see "holdout problems" in economics). counter-coordination. the fact that a system is decentralized makes it easy for participants not participating in the collusion to make a fork that strips out the colluding attackers and continue the system from there. barriers for users to join the fork are low, and the intention of decentralization creates moral pressure in favor of participating in the fork. risk of defection. it still is much harder for five companies to join together to do something widely considered to be bad than it is for them to join together for a non-controversial or benign purpose. the five companies do not know each other too well, so there is a risk that one of them will refuse to participate and blow the whistle quickly, and the participants have a hard time judging the risk. individual employees within the companies may blow the whistle too. taken together, these barriers are substantial indeed often substantial enough to stop potential attacks in their tracks, even when those five companies are simultaneously perfectly capable of quickly coordinating to do something legitimate. ethereum blockchain miners, for example, are perfectly capable of coordinating increases to the gas limit, but that does not mean that they can so easily collude to attack the chain. the blockchain experience shows how designing protocols as institutionally decentralized architectures, even when it's well-known ahead of time that the bulk of the activity will be dominated by a few companies, can often be a very valuable thing. this idea is not limited to blockchains; it can be applied in other contexts as well (eg. see here for applications to antitrust). forking as counter-coordination but we cannot always effectively prevent harmful collusions from taking place. and to handle those cases where a harmful collusion does take place, it would be nice to make systems that are more robust against them more expensive for those colluding, and easier to recover for the system. there are two core operating principles that we can use to achieve this end: (1) supporting counter-coordination and (2) skin-in-the-game. the idea behind counter-coordination is this: we know that we cannot design systems to be passively robust to collusions, in large part because there is an extremely large number of ways to organize a collusion and there is no passive mechanism that can detect them, but what we can do is actively respond to collusions and strike back. in digital systems such as blockchains (this could also be applied to more mainstream systems, eg. dns), a major and crucially important form of counter-coordination is forking. if a system gets taken over by a harmful coalition, the dissidents can come together and create an alternative version of the system, which has (mostly) the same rules except that it removes the power of the attacking coalition to control the system. forking is very easy in an open-source software context; the main challenge in creating a successful fork is usually gathering the legitimacy (game-theoretically viewed as a form of "common knowledge") needed to get all those who disagree with the main coalition's direction to follow along with you. this is not just theory; it has been accomplished successfully, most notably in the steem community's rebellion against a hostile takeover attempt, leading to a new blockchain called hive in which the original antagonists have no power. markets and skin in the game another class of collusion-resistance strategy is the idea of skin in the game. skin in the game, in this context, basically means any mechanism that holds individual contributors in a decision individually accountable for their contributions. if a group makes a bad decision, those who approved the decision must suffer more than those who attempted to dissent. this avoids the "tragedy of the commons" inherent in voting systems. forking is a powerful form of counter-coordination precisely because it introduces skin in the game. in hive, the community fork of steem that threw off the hostile takeover attempt, the coins that were used to vote in favor of the hostile takeover were largely deleted in the new fork. the key individuals who participated in the attack individually suffered as a result. markets are in general very powerful tools precisely because they maximize skin in the game. decision markets (prediction markets used to guide decisions; also called futarchy) are an attempt to extend this benefit of markets to organizational decision-making. that said, decision markets can only solve some problems; in particular, they cannot tell us what variables we should be optimizing for in the first place. structuring coordination this all leads us to an interesting view of what it is that people building social systems do. one of the goals of building an effective social system is, in large part, determining the structure of coordination: which groups of people and in what configurations can come together to further their group goals, and which groups cannot? different coordination structures, different outcomes sometimes, more coordination is good: it's better when people can work together to collectively solve their problems. at other times, more coordination is dangerous: a subset of participants could coordinate to disenfranchise everyone else. and at still other times, more coordination is necessary for another reason: to enable the broader community to "strike back" against a collusion attacking the system. in all three of those cases, there are different mechanisms that can be used to achieve these ends. of course, it is very difficult to prevent communication outright, and it is very difficult to make coordination perfect. but there are many options in between that can nevertheless have powerful effects. here are a few possible coordination structuring techniques: technologies and norms that protect privacy technological means that make it difficult to prove how you behaved (secret ballots, maci and similar tech) deliberate decentralization, distributing control of some mechanism to a wide group of people that are known to not be well-coordinated decentralization in physical space, separating out different functions (or different shares of the same function) to different locations (eg. see samo burja on connections between urban decentralization and political decentralization) decentralization between role-based constituencies, separating out different functions (or different shares of the same function) to different types of participants (eg. in a blockchain: "core developers", "miners", "coin holders", "application developers", "users") schelling points, allowing large groups of people to quickly coordinate around a single path forward. complex schelling points could potentially even be implemented in code (eg. recovery from 51% attacks can benefit from this). speaking a common language (or alternatively, splitting control between multiple constituencies who speak different languages) using per-person voting instead of per-(coin/share) voting to greatly increase the number of people who would need to collude to affect a decision encouraging and relying on defectors to alert the public about upcoming collusions none of these strategies are perfect, but they can be used in various contexts with differing levels of success. additionally, these techniques can and should be combined with mechanism design that attempts to make harmful collusions less profitable and more risky to the extent possible; skin in the game is a very powerful tool in this regard. which combination works best ultimately depends on your specific use case. dark mode toggle base layers and functionality escape velocity 2019 dec 26 see all posts one common strand of thinking in blockchain land goes as follows: blockchains should be maximally simple, because they are a piece of infrastructure that is difficult to change and would lead to great harms if it breaks, and more complex functionality should be built on top, in the form of layer 2 protocols: state channels, plasma, rollup, and so forth. layer 2 should be the site of ongoing innovation, layer 1 should be the site of stability and maintenance, with large changes only in emergencies (eg. a one-time set of serious breaking changes to prevent the base protocol's cryptography from falling to quantum computers would be okay). this kind of layer separation is a very nice idea, and in the long term i strongly support this idea. however, this kind of thinking misses an important point: while layer 1 cannot be too powerful, as greater power implies greater complexity and hence greater brittleness, layer 1 must also be powerful enough for the layer 2 protocols-on-top that people want to build to actually be possible in the first place. once a layer 1 protocol has achieved a certain level of functionality, which i will term "functionality escape velocity", then yes, you can do everything else on top without further changing the base. but if layer 1 is not powerful enough, then you can talk about filling in the gap with layer 2 systems, but the reality is that there is no way to actually build those systems, without reintroducing a whole set of trust assumptions that the layer 1 was trying to get away from. this post will talk about some of what this minimal functionality that constitutes "functionality escape velocity" is. a programming language it must be possible to execute custom user-generated scripts on-chain. this programming language can be simple, and actually does not need to be high-performance, but it needs to at least have the level of functionality required to be able to verify arbitrary things that might need to be verified. this is important because the layer 2 protocols that are going to be built on top need to have some kind of verification logic, and this verification logic must be executed by the blockchain somehow. you may have heard of turing completeness; the "layman's intuition" for the term being that if a programming language is turing complete then it can do anything that a computer theoretically could do. any program in one turing-complete language can be translated into an equivalent program in any other turing-complete language. however, it turns out that we only need something slightly lighter: it's okay to restrict to programs without loops, or programs which are guaranteed to terminate in a specific number of steps. rich statefulness it doesn't just matter that a programming language exists, it also matters precisely how that programming language is integrated into the blockchain. among the more constricted ways that a language could be integrated is if it is used for pure transaction verification: when you send coins to some address, that address represents a computer program p which would be used to verify a transaction that sends coins from that address. that is, if you send a transaction whose hash is h, then you would supply a signature s, and the blockchain would run p(h, s), and if that outputs true then the transaction is valid. often, p is a verifier for a cryptographic signature scheme, but it could do more complex operations. note particularly that in this model p does not have access to the destination of the transaction. however, this "pure function" approach is not enough. this is because this pure function-based approach is not powerful enough to implement many kinds of layer 2 protocols that people actually want to implement. it can do channels (and channel-based systems like the lightning network), but it cannot implement other scaling techniques with stronger properties, it cannot be used to bootstrap systems that do have more complicated notions of state, and so forth. to give a simple example of what the pure function paradigm cannot do, consider a savings account with the following feature: there is a cryptographic key k which can initiate a withdrawal, and if a withdrawal is initiated, within the next 24 hours that same key k can cancel the withdrawal. if a withdrawal remains uncancelled within 24 hours, then anyone can "poke" the account to finalize that withdrawal. the goal is that if the key is stolen, the account holder can prevent the thief from withdrawing the funds. the thief could of course prevent the legitimate owner from getting the funds, but the attack would not be profitable for the thief and so they would probably not bother with it (see the original paper for an explanation of this technique). unfortunately this technique cannot be implemented with just pure functions. the problem is this: there needs to be some way to move coins from a "normal" state to an "awaiting withdrawal" state. but the program p does not have access to the destination! hence, any transaction that could authorize moving the coins to an awaiting withdrawal state could also authorize just stealing those coins immediately; p can't tell the difference. the ability to change the state of coins, without completely setting them free, is important to many kinds of applications, including layer 2 protocols. plasma itself fits into this "authorize, finalize, cancel" paradigm: an exit from plasma must be approved, then there is a 7 day challenge period, and within that challenge period the exit could be cancelled if the right evidence is provided. rollup also needs this property: coins inside a rollup must be controlled by a program that keeps track of a state root r, and changes from r to r' if some verifier p(r, r', data) returns true but it only changes the state to r' in that case, it does not set the coins free. this ability to authorize state changes without completely setting all coins in an account free, is what i mean by "rich statefulness". it can be implemented in many ways, some utxo-based, but without it a blockchain is not powerful enough to implement most layer 2 protocols, without including trust assumptions (eg. a set of functionaries who are collectively trusted to execute those richly-stateful programs). note: yes, i know that if p has access to h then you can just include the destination address as part of s and check it against h, and restrict state changes that way. but it is possible to have a programming language that is too resource-limited or otherwise restricted to actually do this; and surprisingly this often actually is the case in blockchain scripting languages. sufficient data scalability and low latency it turns out that plasma and channels, and other layer 2 protocols that are fully off-chain have some fundamental weaknesses that prevent them from fully replicating the capabilities of layer 1. i go into this in detail here; the summary is that these protocols need to have a way of adjudicating situations where some parties maliciously fail to provide data that they promised to provide, and because data publication is not globally verifiable (you don't know when data was published unless you already downloaded it yourself) these adjudication games are not game-theoretically stable. channels and plasma cleverly get around this instability by adding additional assumptions, particularly assuming that for every piece of state, there is a single actor that is interested in that state not being incorrectly modified (usually because it represents coins that they own) and so can be trusted to fight on its behalf. however, this is far from general-purpose; systems like uniswap, for example, include a large "central" contract that is not owned by anyone, and so they cannot effectively be protected by this paradigm. there is one way to get around this, which is layer 2 protocols that publish very small amounts of data on-chain, but do computation entirely off-chain. if data is guaranteed to be available, then computation being done off-chain is okay, because games for adjudicating who did computation correctly and who did it incorrectly are game-theoretically stable (or could be replaced entirely by snarks or starks). this is the logic behind zk rollup and optimistic rollup. if a blockchain allows for the publication and guarantees the availability of a reasonably large amount of data, even if its capacity for computation remains very limited, then the blockchain can support these layer-2 protocols and achieve a high level of scalability and functionality. just how much data does the blockchain need to be able to process and guarantee? well, it depends on what tps you want. with a rollup, you can compress most activity to ~10-20 bytes per transaction, so 1 kb/sec gives you 50-100 tps, 1 mb/sec gives you 50,000-100,000 tps, and so forth. fortunately, internet bandwidth continues to grow quickly, and does not seem to be slowing down the way moore's law for computation is, so increasing scaling for data without increasing computational load is quite a viable path for blockchains to take! note also that it is not just data capacity that matters, it is also data latency (ie. having low block times). layer 2 protocols like rollup (or for that matter plasma) only give any guarantees of security when the data actually is published to chain; hence, the time it takes for data to be reliably included (ideally "finalized") on chain is the time that it takes between when alice sends bob a payment and bob can be confident that this payment will be included. the block time of the base layer sets the latency for anything whose confirmation depends things being included in the base layer. this could be worked around with on-chain security deposits, aka "bonds", at the cost of high capital inefficiency, but such an approach is inherently imperfect because a malicious actor could trick an unlimited number of different people by sacrificing one deposit. conclusions "keep layer 1 simple, make up for it on layer 2" is not a universal answer to blockchain scalability and functionality problems, because it fails to take into account that layer 1 blockchains themselves must have a sufficient level of scalability and functionality for this "building on top" to actually be possible (unless your so-called "layer 2 protocols" are just trusted intermediaries). however, it is true that beyond a certain point, any layer 1 functionality can be replicated on layer 2, and in many cases it's a good idea to do this to improve upgradeability. hence, we need layer 1 development in parallel with layer 2 development in the short term, and more focus on layer 2 in the long term. dark mode toggle the limits to blockchain scalability 2021 may 23 see all posts special thanks to felix lange, martin swende, marius van der wijden and mark tyneway for feedback and review. just how far can you push the scalability of a blockchain? can you really, as elon musk wishes, "speed up block time 10x, increase block size 10x & drop fee 100x" without leading to extreme centralization and compromising the fundamental properties that make a blockchain what it is? if not, how far can you go? what if you change the consensus algorithm? even more importantly, what if you change the technology to introduce features such as zk-snarks or sharding? a sharded blockchain can theoretically just keep adding more shards; is there such a thing as adding too many? as it turns out, there are important and quite subtle technical factors that limit blockchain scaling, both with sharding and without. in many cases there are solutions, but even with the solutions there are limits. this post will go through what many of these issues are. just increase the parameters, and all problems are solved. but at what cost? it's crucial for blockchain decentralization for regular users to be able to run a node at 2:35 am, you receive an emergency call from your partner on the opposite side of the world who helps run your mining pool (or it could be a staking pool). since about 14 minutes ago, your partner tells you, your pool and a few others split off from the chain which still carries 79% of the network. according to your node, the majority chain's blocks are invalid. there's a balance error: the key block appeared to erroneously assign 4.5 million extra coins to an unknown address. an hour later, you're in a telegram chat with the other two small pools who were caught blindsided just as you were, as well as some block explorers and exchanges. you finally see someone paste a link to a tweet, containing a published message. "announcing new on-chain sustainable protocol development fund", the tweet begins. by the morning, arguments on twitter, and on the one community forum that was not censoring the discussion, discussions are everywhere. but by then a significant part of the 4.5 million coins had been converted on-chain to other assets, and billions of dollars of defi transactions had taken place. 79% of the consensus nodes, and all the major block explorers and endpoints for light wallets, were following this new chain. perhaps the new dev fund will fund some development, or perhaps it will just all be embezzled by the leading pools and exchanges and their cronies. but regardless of how it turns out, the fund is for all intents and purposes a fait accompli, and regular users have no way to fight back. movie coming soon. maybe it can be funded by molochdao or something. can this happen on your blockchain? the elites of your blockchain community, including pools, block explorers and hosted nodes, are probably quite well-coordinated; quite likely they're all in the same telegram channels and wechat groups. if they really want to organize a sudden change to the protocol rules to further their own interests, then they probably can. the ethereum blockchain has fully resolved consensus failures in ten hours; if your blockchain has only one client implementation, and you only need to deploy a code change to a few dozen nodes, coordinating a change to client code can be done much faster. the only reliable way to make this kind of coordinated social attack not effective is through passive defense from the one constituency that actually is decentralized: the users. imagine how the story would have played out if the users were running nodes that verify the chain (whether directly or through more advanced indirect techniques), and automatically reject blocks that break the protocol rules even if over 90% of the miners or stakers support those blocks. if every user ran a verifying node, then the attack would have quickly failed: a few mining pools and exchanges would have forked off and looked quite foolish in the process. but even if some users ran verifying nodes, the attack would not have led to a clean victory for the attacker; rather, it would have led to chaos, with different users seeing different views of the chain. at the very least, the ensuing market panic and likely persistent chain split would greatly reduce the attackers' profits. the thought of navigating such a protracted conflict would itself deter most attacks. listen to hasu on this one. if you have a community of 37 node runners and 80000 passive listeners that check signatures and block headers, the attacker wins. if you have a community where everyone runs a node, the attacker loses. we don't know what the exact threshold is at which herd immunity against coordinated attacks kicks in, but there is one thing that's absolutely clear: more nodes good, fewer nodes bad, and we definitely need more than a few dozen or few hundred. so, what are the limits to how much work we can require full nodes to do? to maximize the number of users who can run a node, we'll focus on regular consumer hardware. there are some increases to capacity that can be achieved by demanding some specialized hardware purchases that are easy to obtain (eg. from amazon), but they actually don't increase scalability by that much. there are three key limitations to a full node's ability to process a large number of transactions: computing power: what % of the cpu can we safely demand to run a node? bandwidth: given the realities of current internet connections, how many bytes can a block contain? storage: how many gigabytes on disk can we require users to store? also, how quickly must it be readable? (ie. is hdd okay or do we need ssd) many erroneous takes on how far a blockchain can scale using "simple" techniques stem from overly optimistic estimates for each of these numbers. we can go through these three factors one by one: computing power bad answer: 100% of cpu power can be spent on block verification correct answer: ~5-10% of cpu power can be spent on block verification there are four key reasons why the limit is so low: we need a safety margin to cover the possibility of dos attacks (transactions crafted by an attacker to take advantage of weaknesses in code to take longer to process than regular transactions) nodes need to be able to sync the chain after being offline. if i drop off the network for a minute, i should be able to catch up in a few seconds running a node should not drain your battery very quickly and make all your other apps very slow there are other non-block-production tasks that nodes need to do as well, mostly around verifying and responding to incoming transactions and requests on the p2p network note that up until recently, most explanations for "why only 5-10%?" focused on a different problem: that because pow blocks come at random times, it taking a long time to verify blocks increases the risk that multiple blocks get created at the same time. there are many fixes to this problem (eg. bitcoin ng, or just using proof of stake). but these fixes do not solve the other four problems, and so they don't enable large gains in scalability as many had initially thought. parallelism is also not a magic bullet. often, even clients of seemingly single-threaded blockchains are parallelized already: signatures can be verified by one thread while execution is done by other threads, and there's a separate thread that's handling transaction pool logic in the background. and the closer you get to 100% usage across all threads, the more energy-draining running a node becomes and the lower your safety margin against dos. bandwidth bad answer: if we have 10 mb blocks every 2-3 seconds, then most users have a >10 mb/sec network, so of course they can handle it correct answer: maybe we can handle 1-5 mb blocks every 12 seconds. it's hard though. nowadays we frequently hear very high advertised statistics for how much bandwidth internet connections can offer: numbers of 100 mbps and even 1 gbps are common to hear. however, there is a large difference between advertised bandwidth and the expected actual bandwidth of a connection for several reasons: "mbps" refers to "millions of bits per second"; a bit is 1/8 of a byte, so you need to divide advertised bit numbers by 8 to get the advertised byte numbers. internet providers, just like all companies, often lie. there's always multiple applications using the same internet connection, so a node can't hog the entire bandwidth. p2p networks inevitably introduce their own overhead: nodes often end up downloading and re-uploading the same block multiple times (not to mention transactions being broadcasted through the mempool before being included in a block). when starkware did an experiment in 2019 where they published 500 kb blocks after the transaction data gas cost decrease made that possible for the first time, a few nodes were actually unable to handle blocks of that size. ability to handle large blocks has since been improved and will continue to be improved. but no matter what we do, we'll still be very far from being able to naively take the average bandwidth in mb/sec, convince ourselves that we're okay with 1s latency, and be able to have blocks that are that size. storage bad answer: 10 terabytes correct answer: 512 gigabytes the main argument here is, as you might guess, the same as elsewhere: the difference between theory and practice. in theory, there are 8 tb solid state drives that you can buy on amazon (you do need ssds or nvme; hdds are too slow for storing the blockchain state). in practice, the laptop that was used to write this blog post has 512 gb, and if you make people go buy their own hardware, many of them will just get lazy (or they can't afford $800 for an 8 tb ssd) and use a centralized provider. and even if you can fit a blockchain onto some storage, a high level of activity can easily quickly burn through the disk and force you to keep getting a new one. a poll in a group of blockchain protocol researchers of how much disk space everyone has. small sample size, i know, but still... additionally, storage size determines the time needed for a new node to be able to come online and start participating in the network. any data that existing nodes have to store is data that a new node has to download. this initial sync time (and bandwidth) is also a major barrier to users being able to run nodes. while writing this blog post, syncing a new geth node took me ~15 hours. if ethereum had 10x more usage, syncing a new geth node would take at least a week, and it would be much more likely to just lead to your internet connection getting throttled. this is all even more important during an attack, when a successful response to the attack will likely involve many users spinning up new nodes when they were not running nodes before. interaction effects additionally, there are interaction effects between these three types of costs. because databases use tree structures internally to store and retrieve data, the cost of fetching data from a database increases with the logarithm of the size of the database. in fact, because the top level (or top few levels) can be cached in ram, the disk access cost is proportional to the size of the database as a multiple of the size of the data cached in ram. don't take this diagram too literally; different databases work in different ways, and often the part in memory is just a single (but big) layer (see lsm trees as used in leveldb). but the basic principles are the same. for example, if the cache is 4 gb, and we assume that each layer of the database is 4x bigger than the previous, then ethereum's current ~64 gb state would require ~2 accesses. but if the state size increases by 4x to ~256 gb, then this would increase to ~3 accesses (so 1.5x more accesses per read). hence, a 4x increase in the gas limit, which would increase both the state size and the number of reads, could actually translate into a ~6x increase in block verification time. the effect may be even stronger: hard disks often take longer to read and write when they are full than when they are near-empty. so what does this mean for ethereum? today in the ethereum blockchain, running a node already is challenging for many users, though it is still at least possible on regular hardware (i just synced a node on my laptop while writing this post!). hence, we are close to hitting bottlenecks. the issue that core developers are most concerned with is storage size. thus, at present, valiant efforts at solving bottlenecks in computation and data, and even changes to the consensus algorithm, are unlikely to lead to large gas limit increases being accepted. even solving ethereum's largest outstanding dos vulnerability only led to a gas limit increase of 20%. the only solution to storage size problems is statelessness and state expiry. statelessness allows for a class of nodes that verify the chain without maintaining permanent storage. state expiry pushes out state that has not been recently accessed, forcing users to manually provide proofs to renew it. both of these paths have been worked at for a long time, and proof-of-concept implementation on statelessness has already started. these two improvements combined can greatly alleviate these concerns and open up room for a significant gas limit increase. but even after statelessness and state expiry are implemented, gas limits may only increase safely by perhaps ~3x until the other limitations start to dominate. another possible medium-term solution is using zk-snarks to verify transactions. zk-snarks would ensure that regular users do not have to personally store the state or verify blocks, though they still would need to download all the data in blocks to protect against data unavailability attacks. additionally, even if attackers cannot force invalid blocks through, if capacity is increased to the point where running a consensus node is too difficult, there is still the risk of coordinated censorship attacks. hence, zk-snarks cannot increase capacity infinitely, but they still can increase capacity by a significant margin (perhaps 1-2 orders of magnitude). some chains are exploring this approach at layer 1; ethereum is getting the benefits of this approach through layer-2 protocols (called zk rollups) such as zksync, loopring and starknet. what happens after sharding? sharding fundamentally gets around the above limitations, because it decouples the data contained on a blockchain from the data that a single node needs to process and store. instead of nodes verifying blocks by personally downloading and executing them, they use advanced mathematical and cryptographic techniques to verify blocks indirectly. as a result, sharded blockchains can safely have very high levels of transaction throughput that non-sharded blockchains cannot. this does require a lot of cryptographic cleverness in creating efficient substitutes for naive full validation that successfully reject invalid blocks, but it can be done: the theory is well-established and proof-of-concepts based on draft specifications are already being worked on. ethereum is planning to use quadratic sharding, where total scalability is limited by the fact that a node has to be able to process both a single shard and the beacon chain which has to perform some fixed amount of management work for each shard. if shards are too big, nodes can no longer process individual shards, and if there are too many shards, nodes can no longer process the beacon chain. the product of these two constraints forms the upper bound. conceivably, one could go further by doing cubic sharding, or even exponential sharding. data availability sampling would certainly become much more complex in such a design, but it can be done. but ethereum is not going further than quadratic. the reason is that the extra scalability gains that you get by going from shards-of-transactions to shards-of-shards-of-transactions actually cannot be realized without other risks becoming unacceptably high. so what are these risks? minimum user count a non-sharded blockchain can conceivably run as long as there is even one user that cares to participate in it. sharded blockchains are not like this: no single node can process the whole chain, and so you need enough nodes so that they can at least process the chain together. if each node can process 50 tps, and the chain can process 10000 tps, then the chain needs at least 200 nodes to survive. if the chain at any point gets to less than 200 nodes, then either nodes stop being able to keep up with the chain, or nodes stop being able to detect invalid blocks, or a number of other bad things may happen, depending on how the node software is set up. in practice, the safe minimum count is several times higher than the naive "chain tps divided by node tps" heuristic due to the need for redundancy (including for data availability sampling); for our above example, let's call it 1000 nodes. if a sharded blockchain's capacity increases by 10x, the minimum user count also increases by 10x. now, you might ask: why don't we start with a little bit of capacity, and increase it only when we see lots of users so we actually need it, and decrease it if the user count goes back down? there are a few problems with this: a blockchain itself cannot reliably detect how many unique users are on it, and so this would require some kind of governance to detect and set the shard count. governance over capacity limits can easily become a locus of division and conflict. what if many users suddenly and unexpectedly drop out at the same time? increasing the minimum number of users needed for a fork to start makes it harder to defend against hostile takeovers. a minimum user count of under 1,000 is almost certainly fine. a minimum user count of 1 million, on the other hand, is certainly not. even a minimum user count of 10,000 is arguably starting to get risky. hence, it seems difficult to justify a sharded blockchain having more than a few hundred shards. history retrievability an important property of a blockchain that users really value is permanence. a digital asset stored on a server will stop existing in 10 years when the company goes bankrupt or loses interest in maintaining that ecosystem. an nft on ethereum, on the other hand, is forever. yes, people will still be downloading and examining your cryptokitties in the year 2371. deal with it. but once a blockchain's capacity gets too high, it becomes harder to store all that data, until at some point there's a large risk that some part of history will just end up being stored by... nobody. quantifying this risk is easy. take the blockchain's data capacity in mb/sec, and multiply by ~30 to get the amount of data stored in terabytes per year. the current sharding plan has a data capacity of ~1.3 mb/sec, so about 40 tb/year. if that is increased by 10x, this becomes 400 tb/year. if we want the data to be not just accessible, but accessible conveniently, we would also need metadata (eg. decompressing rollup transactions), so make that 4 petabytes per year, or 40 petabytes after a decade. the internet archive uses 50 petabytes. so that's a reasonable upper bound for how large a sharded blockchain can safely get. hence, it looks like on both of these dimensions, the ethereum sharding design is actually already roughly targeted fairly close to reasonable maximum safe values. the constants can be increased a little bit, but not too much. summary there are two ways to try to scale a blockchain: fundamental technical improvements, and simply increasing the parameters. increasing the parameters sounds very attractive at first: if you do the math on a napkin, it is easy to convince yourself that a consumer laptop can process thousands of transactions per second, no zk-snarks or rollups or sharding required. unfortunately, there are many subtle reasons why this approach is fundamentally flawed. computers running blockchain nodes cannot spend 100% of cpu power validating the chain; they need a large safety margin to resist unexpected dos attacks, they need spare capacity for tasks like processing transactions in the mempool, and you don't want running a node on a computer to make that computer unusable for any other applications at the same time. bandwidth similarly has overhead: a 10 mb/s connection does not mean you can have a 10 megabyte block every second! a 1-5 megabyte block every 12 seconds, maybe. and it is the same with storage. increasing hardware requirements for running a node and limiting node-running to specialized actors is not a solution. for a blockchain to be decentralized, it's crucially important for regular users to be able to run a node, and to have a culture where running nodes is a common activity. fundamental technical improvements, on the other hand, can work. currently, the main bottleneck in ethereum is storage size, and statelessness and state expiry can fix this and allow an increase of perhaps up to ~3x but not more, as we want running a node to become easier than it is today. sharded blockchains can scale much further, because no single node in a sharded blockchain needs to process every transaction. but even there, there are limits to capacity: as capacity goes up, the minimum safe user count goes up, and the cost of archiving the chain (and the risk that data is lost if no one bothers to archive the chain) goes up. but we don't have to worry too much: those limits are high enough that we can probably process over a million transactions per second with the full security of a blockchain. but it's going to take work to do this without sacrificing the decentralization that makes blockchains so valuable. dark mode toggle crypto cities 2021 oct 31 see all posts special thanks to mr silly and tina zhen for early feedback on the post, and to a big long list of people for discussion of the ideas. one interesting trend of the last year has been the growth of interest in local government, and in the idea of local governments that have wider variance and do more experimentation. over the past year, miami mayor francis suarez has pursued a twitter-heavy tech-startup-like strategy of attracting interest in the city, frequently engaging with the mainstream tech industry and crypto community on twitter. wyoming now has a dao-friendly legal structure, colorado is experimenting with quadratic voting, and we're seeing more and more experiments making more pedestrian-friendly street environments for the offline world. we're even seeing projects with varying degrees of radicalness cul de sac, telosa, citydao, nkwashi, prospera and many more trying to create entire neighborhoods and cities from scratch. another interesting trend of the last year has been the rapid mainstreaming of crypto ideas such as coins, non-fungible tokens and decentralized autonomous organizations (daos). so what would happen if we combine the two trends together? does it make sense to have a city with a coin, an nft, a dao, some record-keeping on-chain for anti-corruption, or even all four? as it turns out, there are already people trying to do just that: citycoins.co, a project that sets up coins intended to become local media of exchange, where a portion of the issuance of the coin goes to the city government. miamicoin already exists, and "san francisco coin" appears to be coming soon. other experiments with coin issuance (eg. see this project in seoul) experiments with nfts, often as a way of funding local artists. busan is hosting a government-backed conference exploring what they could do with nfts. reno mayor hillary schieve's expansive vision for blockchainifying the city, including nft sales to support local art, a renodao with renocoins issued to local residents that could get revenue from the government renting out properties, blockchain-secured lotteries, blockchain voting and more. much more ambitious projects creating crypto-oriented cities from scratch: see citydao, which describes itself as, well, "building a city on the ethereum blockchain" daoified governance and all. but are these projects, in their current form, good ideas? are there any changes that could make them into better ideas? let us find out... why should we care about cities? many national governments around the world are showing themselves to be inefficient and slow-moving in response to long-running problems and rapid changes in people's underlying needs. in short, many national governments are missing live players. even worse, many of the outside-the-box political ideas that are being considered or implemented for national governance today are honestly quite terrifying. do you want the usa to be taken over by a clone of ww2-era portuguese dictator antonio salazar, or perhaps an "american caesar", to beat down the evil scourge of american leftism? for every idea that can be reasonably described as freedom-expanding or democratic, there are ten that are just different forms of centralized control and walls and universal surveillance. now consider local governments. cities and states, as we've seen from the examples at the start of this post, are at least in theory capable of genuine dynamism. there are large and very real differences of culture between cities, so it's easier to find a single city where there is public interest in adopting any particular radical idea than it is to convince an entire country to accept it. there are very real challenges and opportunities in local public goods, urban planning, transportation and many other sectors in the governance of cities that could be addressed. cities have tightly cohesive internal economies where things like widespread cryptocurrency adoption could realistically independently happen. furthermore, it's less likely that experiments within cities will lead to terrible outcomes both because cities are regulated by higher-level governments and because cities have an easier escape valve: people who are unhappy with what's going on can more easily exit. so all in all, it seems like the local level of government is a very undervalued one. and given that criticism of existing smart city initiatives often heavily focuses on concerns around centralized governance, lack of transparency and data privacy, blockchain and cryptographic technologies seem like a promising key ingredient for a more open and participatory way forward. what are city projects up to today? quite a lot actually! each of these experiments is still small scale and largely still trying to find its way around, but they are all at least seeds that could turn into interesting things. many of the most advanced projects are in the united states, but there is interest across the world; over in korea the government of busan is running an nft conference. here are a few examples of what is being done today. blockchain experiments in reno reno, nevada mayor hillary schieve is a blockchain fan, focusing primarily on the tezos ecosystem, and she has recently been exploring blockchain-related ideas (see her podcast here) in the governance of her city: selling nfts to fund local art, starting with an nft of the "space whale" sculpture in the middle of the city creating a reno dao, governed by reno coins that reno residents would be eligible to receive via an airdrop. the reno dao could start to get sources of revenue; one proposed idea was the city renting out properties that it owns and the revenue going into a dao using blockchains to secure all kinds of processes: blockchain-secured random number generators for casinos, blockchain-secured voting, etc. reno space whale. source here. citycoins.co citycoins.co is a project built on stacks, a blockchain run by an unusual "proof of transfer" (for some reason abbreviated pox and not pot) block production algorithm that is built around the bitcoin blockchain and ecosystem. 70% of the coin's supply is generated by an ongoing sale mechanism: anyone with stx (the stacks native token) can send their stx to the city coin contract to generate city coins; the stx revenues are distributed to existing city coin holders who stake their coins. the remaining 30% is made available to the city government. citycoins has made the interesting decision of trying to make an economic model that does not depend on any government support. the local government does not need to be involved in creating a citycoins.co coin; a community group can launch a coin by themselves. an faq-provided answer to "what can i do with citycoins?" includes examples like "citycoins communities will create apps that use tokens for rewards" and "local businesses can provide discounts or benefits to people who ... stack their citycoins". in practice, however, the miamicoin community is not going at it alone; the miami government has already de-facto publicly endorsed it. miamicoin hackathon winner: a site that allows coworking spaces to give preferential offers to miamicoin holders. citydao citydao is the most radical of the experiments: unlike miami and reno, which are existing cities with existing infrastructure to be upgraded and people to be convinced, citydao a dao with legal status under the wyoming dao law (see their docs here) trying to create entirely new cities from scratch. so far, the project is still in its early stages. the team is currently finalizing a purchase of their first plot of land in a far-off corner of wyoming. the plan is to start with this plot of land, and then add other plots of land in the future, to build cities, governed by a dao and making heavy use of radical economic ideas like harberger taxes to allocate the land, make collective decisions and manage resources. their dao is one of the progressive few that is avoiding coin voting governance; instead, the governance is a voting scheme based on "citizen" nfts, and ideas have been floated to further limit votes to one-per-person by using proof-of-humanity verification. the nfts are currently being sold to crowdfund the project; you can buy them on opensea. what do i think cities could be up to? obviously there are a lot of things that cities could do in principle. they could add more bike lanes, they could use co2 meters and far-uvc light to more effectively reduce covid spread without inconveniencing people, and they could even fund life extension research. but my primary specialty is blockchains and this post is about blockchains, so... let's focus on blockchains. i would argue that there are two distinct categories of blockchain ideas that make sense: using blockchains to create more trusted, transparent and verifiable versions of existing processes. using blockchains to implement new and experimental forms of ownership for land and other scarce assets, as well as new and experimental forms of democratic governance. there's a natural fit between blockchains and both of these categories. anything happening on a blockchain is very easy to publicly verify, with lots of ready-made freely available tools to help people do that. any application built on a blockchain can immediately plug in to and interface with other applications in the entire global blockchain ecosystem. blockchain-based systems are efficient in a way that paper is not, and publicly verifiable in a way that centralized computing systems are not a necessary combination if you want to, say, make a new form of voting that allows citizens to give high-volume real-time feedback on hundreds or thousands of different issues. so let's get into the specifics. what are some existing processes that blockchains could make more trusted and transparent? one simple idea that plenty of people, including government officials around the world, have brought up to me on many occasions is the idea of governments creating a whitelisted internal-use-only stablecoin for tracking internal government payments. every tax payment from an individual or organization could be tied to a publicly visible on-chain record minting that number of coins (if we want individual tax payment quantities to be private, there are zero-knowledge ways to make only the total public but still convince everyone that it was computed correctly). transfers between departments could be done "in the clear", and the coins would be redeemed only by individual contractors or employees claiming their payments and salaries. this system could easily be extended. for example, procurement processes for choosing which bidder wins a government contract could largely be done on-chain. many more processes could be made more trustworthy with blockchains: fair random number generators (eg. for lotteries) vdfs, such as the one ethereum is expected to include, could serve as a fair random number generator that could be used to make government-run lotteries more trustworthy. fair randomness could also be used for many other use cases, such as sortition as a form of government. certificates, for example cryptographic proofs that some particular individual is a resident of the city, could be done on-chain for added verifiability and security (eg. if such certificates are issued on-chain, it would become obvious if a large number of false certificates are issued). this can be used by all kinds of local-government-issued certificates. asset registries, for land and other assets, as well as more complicated forms of property ownership such as development rights. due to the need for courts to be able to make assignments in exceptional situations, these registries will likely never be fully decentralized bearer instruments in the same way that cryptocurrencies are, but putting records on-chain can still make it easier to see what happened in what order in a dispute. eventually, even voting could be done on-chain. here, many complexities and dragons loom and it's really important to be careful; a sophisticated solution combining blockchains, zero knowledge proofs and other cryptography is needed to achieve all the desired privacy and security properties. however, if humanity is ever going to move to electronic voting at all, local government seems like the perfect place to start. what are some radical economic and governance experiments that could be interesting? but in addition to these kinds of blockchain overlays onto things that governments already do, we can also look at blockchains as an opportunity for governments to make completely new and radical experiments in economics and governance. these are not necessarily final ideas on what i think should be done; they are more initial explorations and suggestions for possible directions. once an experiment starts, real-world feedback is often by far the most useful variable to determine how the experiment should be adjusted in the future. experiment #1: a more comprehensive vision of city tokens citycoins.co is one vision for how city tokens could work. but it is far from the only vision. indeed, the citycoins.so approach has significant risks, particularly in how economic model is heavily tilted toward early adopters. 70% of the stx revenue from minting new coins is given to existing stakers of the city coin. more coins will be issued in the next five years than in the fifty years that follow. it's a good deal for the government in 2021, but what about 2051? once a government endorses a particular city coin, it becomes difficult for it to change directions in the future. hence, it's important for city governments to think carefully about these issues, and choose a path that makes sense for the long term. here is a different possible sketch of a narrative of how city tokens might work. it's far from the only possible alternative to the citycoins.co vision; see steve waldman's excellent article arguing for a city-localized medium of exchange for yet another possible direction. in any case, city tokens are a wide design space, and there are many different options worth considering. anyway, here goes... the concept of home ownership in its current form is a notable double-edged sword, and the specific ways in which it's actively encouraged and legally structured is considered by many to be one of the biggest economic policy mistakes that we are making today. there is an inevitable political tension between a home as a place to live and a home as an investment asset, and the pressure to satisfy communities who care about the latter often ends up severely harming the affordability of the former. a resident in a city either owns a home, making them massively over-exposed to land prices and introducing perverse incentives to fight against construction of new homes, or they rent a home, making them negatively exposed to the real estate market and thus putting them economically at odds with the goal of making a city a nice place to live. but even despite all of these problems, many still find home ownership to be not just a good personal choice, but something worthy of actively subsidizing or socially encouraging. one big reason is that it nudges people to save money and build up their net worth. another big reason is that despite its flaws, it creates economic alignment between residents and the communities they live in. but what if we could give people a way to save and create that economic alignment without the flaws? what if we could create a divisible and fungible city token, that residents could hold as many units of as they can afford or feel comfortable with, and whose value goes up as the city prospers? first, let's start with some possible objectives. not all are necessary; a token that accomplishes only three of the five is already a big step forward. but we'll try to hit as many of them as possible: get sustainable sources of revenue for the government. the city token economic model should avoid redirecting existing tax revenue; instead, it should find new sources of revenue. create economic alignment between residents and the city. this means first of all that the coin itself should clearly become more valuable as the city becomes more attractive. but it also means that the economics should actively encourage residents to hold the coin more than faraway hedge funds. promote saving and wealth-building. home ownership does this: as home owners make mortgage payments, they build up their net worth by default. city tokens could do this too, making it attractive to accumulate coins over time, and even gamifying the experience. encourage more pro-social activity, such as positive actions that help the city and more sustainable use of resources. be egalitarian. don't unduly favor wealthy people over poor people (as badly designed economic mechanisms often do accidentally). a token's divisibility, avoiding a sharp binary divide between haves and have-nots, does a lot already, but we can go further, eg. by allocating a large portion of new issuance to residents as a ubi. one pattern that seems to easily meet the first three objectives is providing benefits to holders: if you hold at least x coins (where x can go up over time), you get some set of services for free. miamicoin is trying to encourage businesses to do this, but we could go further and make government services work this way too. one simple example would be making existing public parking spaces only available for free to those who hold at least some number of coins in a locked-up form. this would serve a few goals at the same time: create an incentive to hold the coin, sustaining its value. create an incentive specifically for residents to hold the coin, as opposed to otherwise-unaligned faraway investors. furthermore, the incentive's usefulness is capped per-person, so it encourages widely distributed holdings. creates economic alignment (city becomes more attractive -> more people want to park -> coins have more value). unlike home ownership, this creates alignment with an entire town, and not merely a very specific location in a town. encourage sustainable use of resources: it would reduce usage of parking spots (though people without coins who really need them could still pay), supporting many local governments' desires to open up more space on the roads to be more pedestrian-friendly. alternatively, restaurants could also be allowed to lock up coins through the same mechanism and claim parking spaces to use for outdoor seating. but to avoid perverse incentives, it's extremely important to avoid overly depending on one specific idea and instead to have a diverse array of possible revenue sources. one excellent gold mine of places to give city tokens value, and at the same time experiment with novel governance ideas, is zoning. if you hold at least y coins, then you can quadratically vote on the fee that nearby landowners have to pay to bypass zoning restrictions. this hybrid market + direct democracy based approach would be much more efficient than current overly cumbersome permitting processes, and the fee itself would be another source of government revenue. more generally, any of the ideas in the next section could be combined with city tokens to give city token holders more places to use them. experiment #2: more radical and participatory forms of governance this is where radical markets ideas such as harberger taxes, quadratic voting and quadratic funding come in. i already brought up some of these ideas in the section above, but you don't have to have a dedicated city token to do them. some limited government use of quadratic voting and funding has already happened: see the colorado democratic party and the taiwanese presidential hackathon, as well as not-yet-government-backed experiments like gitcoin's boulder downtown stimulus. but we could do more! one obvious place where these ideas can have long-term value is giving developers incentives to improve the aesthetics of buildings that they are building (see here, here, here and here for some recent examples of professional blabbers debating the aesthetics of modern architecture). harberger taxes and other mechanisms could be used to radically reform zoning rules, and blockchains could be used to administer such mechanisms in a more trustworthy and efficient way. another idea that is more viable in the short term is subsidizing local businesses, similar to the downtown stimulus but on a larger and more permanent scale. businesses produce various kinds of positive externalities in their local communities all the time, and those externalities could be more effectively rewarded. local news could be quadratically funded, revitalizing a long-struggling industry. pricing for advertisements could be set based on real-time votes of how much people enjoy looking at each particular ad, encouraging more originality and creativity. more democratic feedback (and possibly even retroactive democratic feedback!) could plausibly create better incentives in all of these areas. and 21st-century digital democracy through real-time online quadratic voting and funding could plausibly do a much better job than 20th-century democracy, which seems in practice to have been largely characterized by rigid building codes and obstruction at planning and permitting hearings. and of course, if you're going to use blockchains to secure voting, starting off by doing it with fancy new kinds of votes seems far more safe and politically feasible than re-fitting existing voting systems. mandatory solarpunk picture intended to evoke a positive image of what might happen to our cities if real-time quadratic votes could set subsidies and prices for everything. conclusions there are a lot of worthwhile ideas for cities to experiment with that could be attempted by existing cities or by new cities. new cities of course have the advantage of not having existing residents with existing expectations of how things should be done; but the concept of creating a new city itself is, in modern times, relatively untested. perhaps the multi-billion-dollar capital pools in the hands of people and projects enthusiastic to try new things could get us over the hump. but even then, existing cities will likely continue to be the place where most people live for the foreseeable future, and existing cities can use these ideas too. blockchains can be very useful in both the more incremental and more radical ideas that were proposed here, even despite the inherently "trusted" nature of a city government. running any new or existing mechanism on-chain gives the public an easy ability to verify that everything is following the rules. public chains are better: the benefits from existing infrastructure for users to independently verify what is going on far outweigh the losses from transaction fees, which are expected to quickly decrease very soon from rollups and sharding. if strong privacy is required, blockchains can be combined zero knowledge cryptography to give privacy and security at the same time. the main trap that governments should avoid is too quickly sacrificing optionality. an existing city could fall into this trap by launching a bad city token instead of taking things more slowly and launching a good one. a new city could fall into this trap by selling off too much land, sacrificing the entire upside to a small group of early adopters. starting with self-contained experiments, and taking things slowly on moves that are truly irreversible, is ideal. but at the same time, it's also important to seize the opportunity in the first place. there's a lot that can and should be improved with cities, and a lot of opportunities; despite the challenges, crypto cities broadly are an idea whose time has come. dark mode toggle where to use a blockchain in non-financial applications? 2022 jun 12 see all posts special thanks to shrey jain and puja ohlhaver for substantial feedback and review recently, there has been a growing amount of interest in using blockchains for not-just-financial applications. this is a trend that i have been strongly in favor of, for various reasons. in the last month, puja ohlhaver, glen weyl and i collaborated on a paper describing a more detailed vision for what could be done with a richer ecosystem of soulbound tokens making claims describing various kinds of relationships. this has led to some discussion, particularly focused on whether or not it makes any sense to use a blockchain in a decentralized identity ecosystem: kate sills argues for off-chain signed claims puja ohlhaver responds to kate sills evin mcmullen and myself have a podcast debating on-chain vs off-chain attestations kevin yu writes a technical overview bringing up the on-chain versus off-chain question molly white argues a pessimistic case against self-sovereign identity shrey jain makes a meta-thread containing the above and many other twitter discussions it's worth zooming out and asking a broader question: where does it make sense, in general, to use a blockchain in non-financial applications? should we move toward a world where even decentralized chat apps work by every message being an on-chain transaction containing the encrypted message? or, alternatively, are blockchains only good for finance (say, because network effects mean that money has a unique need for a "global view"), with all other applications better done using centralized or more local systems? my own view tends to be, like with blockchain voting, far from the "blockchain everywhere" viewpoint, but also far from a "blockchain minimalist". i see the value of blockchains in many situations, sometimes for really important goals like trust and censorship resistance but sometimes purely for convenience. this post will attempt to describe some types of situations where blockchains might be useful, especially in the context of identity, and where they are not. this post is not a complete list and intentionally leaves many things out. the goal is rather to elucidate some common categories. user account key changes and recovery one of the biggest challenges in a cryptographic account system is the issue of key changes. this can happen in a few cases: you're worried that your current key might get lost or stolen, and you want to switch to a different key you want to switch to a different cryptographic algorithm (eg. because you're worried quantum computers will come soon and you want to upgrade to post-quantum) your key got lost, and you want to regain access to your account your key got stolen, and you want to regain exclusive access to your account (and you don't want the thief to be able to do the same) [1] and [2] are relatively simple in that they can be done in a fully self-sovereign way: you control key x, you want to switch to key y, so you publish a message signed with x saying "authenticate me with y from now on", and everyone accepts that. but notice that even for these simpler key change scenarios, you can't just use cryptography. consider the following sequence of events: you are worried that key a might get stolen, so you sign a message with a saying "i use b now" a year later, a hacker actually does steal key a. they sign a message saying with a saying "i use c now", where c is their own key from the point of view of someone coming in later who just receives these two messages, they see that a is no longer used, but they don't know whether "replace a with b" or "replace a with c" has higher priority. this is equivalent to the famous double-spend problem in designing decentralized currencies, except instead of the goal being to prevent a previous owner of a coin from being able to send it again, here the goal is to prevent the previous key controlling an account from being able to change the key. just like creating a decentralized currency, doing account management in a decentralized way requires something like a blockchain. a blockchain can timestamp the key change messages, providing common knowledge over whether b or c came first. [3] and [4] are harder. in general, my own preferred solution is multisig and social recovery wallets, where a group of friends, family members and other contacts can transfer control of your account to a new key if it gets lost or stolen. for critical operations (eg. transferring large quantities of funds, or signing an important contract), participation of this group can also be required. but this too requires a blockchain. social recovery using secret sharing is possible, but it is more difficult in practice: if you no longer trust some of your contacts, or if they want to change their own keys, you have no way to revoke access without changing your key yourself. and so we're back to requiring some form of on-chain record. one subtle but important idea in the desoc paper is that to preserve non-transferability, social recovery (or "community recovery") of profiles might actually need to be mandatory. that is, even if you sell your account, you can always use community recovery to get the account back. this would solve problems like not-actually-reputable drivers buying verified accounts on ride sharing platforms. that said, this is a speculative idea and does not have to be fully implemented to get the other benefits of blockchain-based identity and reputation systems. note that so far this is a limited use-case of blockchains: it's totally okay to have accounts on-chain but do everything else off-chain. there's a place for these kinds of hybrid visions; sign-in with ethereum is good simple example of how this could be done in practice. modifying and revoking attestations alice goes to example college and gets a degree in example studies. she gets a digital record certifying this, signed with example college's keys. unfortunately, six months later, example college discovers that alice had committed a large amount of plagiarism, and revokes her degree. but alice continues to use her old digital record to go around claiming to various people and institutions that she has a degree. potentially, the attestation could even carry permissions for example, the right to log in to the college's online forum and alice might try to inappropriately access that too. how do we prevent this? the "blockchain maximalist" approach would be to make the degree an on-chain nft, so example college can then issue an on-chain transaction to revoke the nft. but perhaps this is needlessly expensive: issuance is common, revocation is rare, and we don't want to require example college to issue transactions and pay fees for every issuance if they don't have to. so instead we can go with a hybrid solution: make initial degree an off-chain signed message, and do revocations on-chain. this is the approach that opencerts uses. the fully off-chain solution, and the one advocated by many off-chain verifiable credentials proponents, is that example college runs a server where they publish a full list of their revocations (to improve privacy, each attestation can come with an attached nonce and the revocation list can just be a list of nonces). for a college, running a server is not a large burden. but for any smaller organization or individual, managing "yet another server script" and making sure it stays online is a significant burden for it people. if we tell people to "just use a server" out of blockchain-phobia, then the likely outcome is that everyone outsources the task to a centralized provider. better to keep the system decentralized and just use a blockchain especially now that rollups, sharding and other techniques are finally starting to come online to make the cost of a blockchain cheaper and cheaper. negative reputation another important area where off-chain signatures do not suffice is negative reputation that is, attestations where the person or organization that you're making attestations about might not want you to see them. i'm using "negative reputation" here as a technical term: the most obvious motivating use case is attestations saying bad things about someone, like a bad review or a report that someone acted abusively in some context, but there are also use cases where "negative" attestations don't imply bad behavior for example, taking out a loan and wanting to prove that you have not taken out too many other loans at the same time. with off-chain claims, you can do positive reputation, because it's in the interest of the recipient of a claim to show it to appear more reputable (or make a zk-proof about it), but you can't do negative reputation, because someone can always choose to only show the claims that make them look good and leave out all the others. here, making attestations on-chain actually does fix things. to protect privacy, we can add encryption and zero knowledge proofs: an attestation can just be an on-chain record with data encrypted to the recipient's public key, and users could prove lack of negative reputation by running a zero knowledge proof that walks over the entire history of records on chain. the proofs being on-chain and the verification process being blockchain-aware makes it easy to verify that the proof actually did walk over the whole history and did not skip any records. to make this computationally feasible, a user could use incrementally verifiable computation (eg. halo) to maintain and prove a tree of records that were encrypted to them, and then reveal parts of the tree when needed. negative reputation and revoking attestations are in some sense equivalent problems: you can revoke an attestation by adding another negative-reputation attestation saying "this other attestation doesn't count anymore", and you can implement negative reputation with revocation by piggybacking on positive reputation: alice's degree at example college could be revoked and replaced with a degree saying "alice got a degree in example studies, but she took out a loan". is negative reputation a good idea? one critique of negative reputation that we sometimes hear is: but isn't negative reputation a dystopian scheme of "scarlet letters", and shouldn't we try our best to do things with positive reputation instead? here, while i support the goal of avoiding unlimited negative reputation, i disagree with the idea of avoiding it entirely. negative reputation is important for many use cases. uncollateralized lending, which is highly valuable for improving capital efficiency within the blockchain space and outside, clearly benefits from it. unirep social shows a proof-of-concept social media platform that combines a high level of anonymity with a privacy-preserving negative reputation system to limit abuse. sometimes, negative reputation can be empowering and positive reputation can be exclusionary. an online forum where every unique human gets the right to post until they get too many "strikes" for misbehavior is more egalitarian than a forum that requires some kind of "proof of good character" to be admitted and allowed to speak in the first place. marginalized people whose lives are mostly "outside the system", even if they actually are of good character, would have a hard time getting such proofs. readers of the strong civil-libertarian persuasion may also want to consider the case of an anonymous reputation system for clients of sex workers: you want to protect privacy, but you also might want a system where if a client mistreats a sex worker, they get a "black mark" that encourages other workers to be more careful or stay away. in this way, negative reputation that's hard to hide can actually empower the vulnerable and protect safety. the point here is not to defend some specific scheme for negative reputation; rather, it's to show that there's very real value that negative reputation unlocks, and a successful system needs to support it somehow. negative reputation does not have to be unlimited negative reputation: i would argue that it should always be possible to create a new profile at some cost (perhaps sacrificing a lot or all of your existing positive reputation). there is a balance between too little accountability and too much accountability. but having some technology that makes negative reputation possible in the first place is a prerequisite for unlocking this design space. committing to scarcity another example of where blockchains are valuable is issuing attestations that have a provably limited quantity. if i want to make an endorsement for someone (eg. one might imagine a company looking for jobs or a government visa program looking at such endorsements), the third party looking at the endorsement would want to know whether i'm careful with endorsements or if i give them off to pretty much whoever is friends with me and asks nicely. the ideal solution to this problem would be to make endorsements public, so that endorsements become incentive-aligned: if i endorse someone who turns out to do something wrong, everyone can discount my endorsements in the future. but often, we also want to preserve privacy. so instead what i could do is publish hashes of each endorsement on-chain, so that anyone can see how many i have given out. an even more effective usecase is many-at-a-time issuance: if an artists wants to issue n copies of a "limited-edition" nft, they could publish on-chain a single hash containing the merkle root of the nfts that they are issuing. the single issuance prevents them from issuing more after the fact, and you can publish the number (eg. 100) signifying the quantity limit along with the merkle root, signifying that only the leftmost 100 merkle branches are valid. by publishing a single merkle root and max count on-chain, you can commit issue a limited quantity of attestations. in this example, there are only five possible valid merkle branches that could satisfy the proof check. astute readers may notice a conceptual similarity to plasma chains. common knowledge one of the powerful properties of blockchains is that they create common knowledge: if i publish something on-chain, then alice can see it, alice can see that bob can see it, charlie can see that alice can see that bob can see it, and so on. common knowledge is often important for coordination. for example, a group of people might want to speak out about an issue, but only feel comfortable doing so if there's enough of them speaking out at the same time that they have safety in numbers. one possible way to do this is for one person to start a "commitment pool" around a particular statement, and invite others to publish hashes (which are private at first) denoting their agreement. only if enough people participate within some period of time, all participants would be required to have their next on-chain message publicly reveal their position. a design like this could be accomplished with a combination of zero knowledge proofs and blockchains (it could be done without blockchains, but that requires either witness encryption, which is not yet available, or trusted hardware, which has deeply problematic security assumptions). there is a large design space around these kinds of ideas that is very underexplored today, but could easily start to grow once the ecosystem around blockchains and cryptographic tools grows further. interoperability with other blockchain applications this is an easy one: some things should be on-chain to better interoperate with other on-chain applications. proof of humanity being an on-chain nft makes it easier for projects to automatically airdrop or give governance rights to accounts that have proof of humanity profiles. oracle data being on-chain makes it easier for defi projects to read. in all of these cases, the blockchain does not remove the need for trust, though it can house structures like daos that manage the trust. but the main value that being on-chain provides is simply being in the same place as the stuff that you're interacting with, which needs a blockchain for other reasons. sure, you could run an oracle off-chain and require the data to be imported only when it needs to be read, but in many cases that would actually be more expensive, and needlessly impose complexity and costs on developers. open-source metrics one key goal of the decentralized society paper is the idea that it should be possible to make calculations over the graph of attestations. a really important one is measuring decentralization and diversity. for example, many people seem to agree that an ideal voting mechanism would somehow keep diversity in mind, giving greater weight to projects that are supported not just by the largest number of coins or even humans, but by the largest number of truly distinct perspectives. quadratic funding as implemented in gitcoin grants also includes some explicitly diversity-favoring logic to mitigate attacks. another natural place where measurements and scores are going to be valuable is reputation systems. this already exists in a centralized form with ratings, but it can be done in a much more decentralized way where the algorithm is transparent while at the same time preserving more user privacy. aside from tightly-coupled use cases like this, where attempts to measure to what extent some set of people is connected and feed that directly into a mechanism, there's also broader use case of helping a community understand itself. in the case of measuring decentralization, this might be a matter of identifying areas where concentration is getting too high, which might require a response. in all of these cases, running computerized algorithms over large bodies of attestations and commitments and doing actually important things with the outputs is going to be unavoidable. we should not try to abolish quantified metrics, we should try to make better ones kate sills expressed her skepticism of the goal of making calculations over reputation, an argument that applies both for public analytics and for individuals zk-proving over their reputation (as in unirep social): the process of evaluating a claim is very subjective and context-dependent. people will naturally disagree about the trustworthiness of other people, and trust depends on the context ... [because of this] we should be extremely skeptical of any proposal to "calculate over" claims to get objective results. i this case, i agree with the importance of subjectivity and context, but i would disagree with the more expansive claim that avoiding calculations around reputation entirely is the right goal to be aiming towards. pure individualized analysis does not scale far beyond dunbar's number, and any complex society that is attempting to support large-scale cooperation has to rely on aggregations and simplifications to some extent. that said, i would argue that an open-participation ecosystem of attestations (as opposed to the centralized one we have today) can get us the best of both worlds by opening up space for better metrics. here are some principles that such designs could follow: inter-subjectivity: eg. a reputation should not be a single global score; instead, it should be a more subjective calculation involving the person or entity being evaluated but also the viewer checking the score, and potentially even other aspects of the local context. credible neutrality: the scheme should clearly not leave room for powerful elites to constantly manipulate it in their own favor. some possible ways to achieve this are maximum transparency and infrequent change of the algorithm. openness: the ability to make meaningful inputs, and to audit other people's outputs by running the check yourself, should be open to anyone, and not just restricted to a small number of powerful groups. if we don't create good large-scale aggregates of social data, then we risk ceding market share to opaque and centralized social credit scores instead. not all data should be on-chain, but making some data public in a common-knowledge way can help increase a community's legibility to itself without creating data-access disparities that could be abused to centralize control. as a data store this is the really controversial use case, even among those who accept most of the others. there is a common viewpoint in the blockchain space that blockchains should only be used in those cases where they are truly needed and unavoidable, and everywhere else we should use other tools. this attitude makes sense in a world where transaction fees are very expensive, and blockchains are uniquely incredibly inefficient. but it makes less sense in a world where blockchains have rollups and sharding and transaction fees have dropped down to a few cents, and the difference in redundancy between a blockchain and non-blockchain decentralized storage might only be 100x. even in such a world, it would not make sense to store all data on-chain. but small text records? absolutely. why? because blockchains are just a really convenient place to store stuff. i maintain a copy of this blog on ipfs. but uploading to ipfs often takes an hour, it requires centralized gateways for users to access it with anything close to website levels of latency, and occasionally files drop off and no longer become visible. dumping the entire blog on-chain, on the other hand, would solve that problem completely. of course, the blog is too big to actually be dumped on-chain, even post-sharding, but the same principle applies to smaller records. some examples of small cases where putting data on-chain just to store it may be the right decision include: augmented secret sharing: splitting your password into n pieces where any m = n-r of the pieces can recover the password, but in a way where you can choose the contents of all n of the pieces. for example, the pieces could all be hashes of passwords, secrets generated through some other tool, or answers to security questions. this is done by publishing an extra r pieces (which are random-looking) on-chain, and doing n-of-(n+r) secret sharing on the whole set. ens optimization. ens could be made more efficient by combining all records into a single hash, only publishing the hash on-chain, and requiring anyone accessing the data to get the full data off of ipfs. but this would significantly increase complexity, and add yet another software dependency. and so ens keeps data on-chain even if it is longer than 32 bytes. social metadata data connected to your account (eg. for sign-in-with-ethereum purposes) that you want to be public and that is very short in length. this is generally not true for larger data like profile pictures (though if the picture happens to be a small svg file it could be!), but it is true for text records. attestations and access permissions. especially if the data being stored is less than a few hundred bytes long, it might be more convenient to store the data on-chain than put the hash on-chain and the data off-chain. in a lot of these cases, the tradeoff isn't just cost but also privacy in those edge cases where keys or cryptography break. sometimes, privacy is only somewhat important, and the occasional loss of privacy from leaked keys or the faraway specter of quantum computing revealing everything in 30 years is less important than having a very high degree of certainty that the data will remain accessible. after all, off-chain data stored in your "data wallet" can get hacked too. but sometimes, data is particularly sensitive, and that can be another argument against putting it on-chain, and keeping it stored locally as a second layer of defense. but note that in those cases, that privacy need is an argument not just against blockchains, but against all decentralized storage. conclusions out of the above list, the two i am personally by far the most confident about are interoperability with other blockchain applications and account management. the first is on-chain already, and the second is relatively cheap (need to use the chain once per user, and not once per action), the case for it is clear, and there really isn't a good non-blockchain-based solution. negative reputation and revocations are also important, though they are still relatively early-stage use cases. a lot can be done with reputation by relying on off-chain positive reputation only, but i expect that the case for revocation and negative reputation will become more clear over time. i expect there to be attempts to do it with centralized servers, but over time it should become clear that blockchains are the only way to avoid a hard choice between inconvenience and centralization. blockchains as data stores for short text records may be marginal or may be significant, but i do expect at least some of that kind of usage to keep happening. blockchains really are just incredibly convenient for cheap and reliable data retrieval, where data continues to be retrievable whether the application has two users or two million. open-source metrics are still a very early-stage idea, and it remains to see just how much can be done and made open without it becoming exploitable (as eg. online reviews, social media karma and the like get exploited all the time). and common knowledge games require convincing people to accept entirely new workflows for socially important things, so of course that is an early-stage idea too. i have a large degree of uncertainty in exactly what level of non-financial blockchain usage in each of these categories makes sense, but it seems clear that blockchains as an enabling tool in these areas should not be dismissed. dark mode toggle endnotes on 2020: crypto and beyond 2020 dec 28 see all posts i'm writing this sitting in singapore, the city in which i've now spent nearly half a year of uninterrupted time an unremarkable duration for many, but for myself the longest i've stayed in any one place for nearly a decade. after months of fighting what may perhaps even be humanity's first boss-level enemy since 1945, the city itself is close to normal, though the world as a whole and its 7.8 billion inhabitants, normally so close by, continue to be so far away. many other parts of the world have done less well and suffered more, though there is now a light at the end of the tunnel, as hopefully rapid deployment of vaccines will help humanity as a whole overcome this great challenge. 2020 has been a strange year because of these events, but also others. as life "away from keyboard (afk)" has gotten much more constrained and challenging, the internet has been supercharged, with consequences both good and bad. politics around the world has gone in strange directions, and i am continually worried by the many political factions that are so easily abandoning their most basic principles because they seem to have decided that their (often mutually contradictory) personal causes are just too important. and yet at the same time, there are rays of hope coming from unusual corners, with new technological discoveries in transportation, medicine, artificial intelligence and, of course, blockchains and cryptography that could open up a new chapter for humanity finally coming to fruition. where we started where we're going and so, 2020 is as good a year as any to ponder a key question: how should we re-evaluate our models of the world? what ways of seeing, understanding and reasoning about the world are going to be more useful in the decades to come, and what paths are no longer as valuable? what paths did we not see before that were valuable all along? in this post, i will give some of my own answers, covering far from everything but digging into a few specifics that seem particularly interesting. it's sometimes hard to tell which of these ideas are a recognition of a changing reality, and which are just myself finally seeing what has always been there; often enough it's some of both. the answers to these questions have a deep relevance both to the crypto space that i call home as well as to the wider world. the changing role of economics economics has historically focused on "goods" in the form of physical objects: production of food, manufacturing of widgets, buying and selling houses, and the like. physical objects have some particular properties: they can be transferred, destroyed, bought and sold, but not copied. if one person is using a physical object, it's usually impractical for another person to use it simultaneously. many objects are only valuable if "consumed" outright. making ten copies of an object requires something close to ten times the resources that it takes to make one (not quite ten times, but surprisingly close, especially at larger scales). but on the internet, very different rules apply. copying is cheap. i can write an article or a piece of code once, and it usually takes quite a bit of effort to write it once, but once that work is done, an unlimited number of people can download and enjoy it. very few things are "consumable"; often products are superseded by better ones, but if that does not happen, something produced today may continue to provide value to people until the end of time. on the internet, "public goods" take center stage. certainly, private goods exist, particularly in the form of individuals' scarce attention and time and virtual assets that command that attention, but the average interaction is one-to-many, not one-to-one. confounding the situation even further, the "many" rarely maps easily to our traditional structures for structuring one-to-many interactions, such as companies, cities or countries;. instead, these public goods are typically public across a widely scattered collection of people all around the world. many online platforms serving wide groups of people need governance, to decide on features, content moderation policies or other challenges important to their user community, though there too, the user community rarely maps cleanly to anything but itself. how is it fair for the us government to govern twitter, when twitter is often a platform for public debates between us politicians and representatives of its geopolitical rivals? but clearly, governance challenges exist and so we need more creative solutions. this is not merely of interest to "pure" online services. though goods in the physical world food, houses, healthcare, transportation continue to be as important as ever, improvements in these goods depend even more than before on technology, and technological progress does happen over the internet. examples of important public goods in the ethereum ecosystem that were funded by the recent gitcoin quadratic funding round. open source software ecosystems, including blockchains, are hugely dependent on public goods. but also, economics itself seems to be a less powerful tool in dealing with these issues. out of all the challenges of 2020, how many can be understood by looking at supply and demand curves? one way to see what is going on here is by looking at the relationship between economics and politics. in the 19th century, the two were frequently viewed as being tied together, a subject called "political economy". in the 20th century, the two are more typically split apart. but in the 21st century, the lines between "private" and "public" are once again rapidly blurring. governments are behaving more like market actors, and corporations are behaving more like governments. we see this merge happening in the crypto space as well, as the researchers' eye of attention is increasingly switching focus to the challenge of governance. five years ago, the main economic topics being considered in the crypto space had to do with consensus theory. this is a tractable economics problem with clear goals, and so we would on several occasions obtain nice clean results like the selfish mining paper. some points of subjectivity, like quantifying decentralization, exist, but they could be easily encapsulated and treated separately from the formal math of the mechanism design. but in the last few years, we have seen the rise of increasingly complicated financial protocols and daos on top of blockchains, and at the same time governance challenges within blockchains. should bitcoin cash redirect 12.5% of its block reward toward paying a developer team? if so, who decides who that developer team is? should zcash extend its 20% developer reward for another four years? these problems certainly can be analyzed economically to some extent, but the analysis inevitably gets stuck at concepts like coordination, flipping between equilibria, "schelling points" and "legitimacy", that are much more difficult to express with numbers. and so, a hybrid discipline, combining formal mathematical reasoning with the softer style of humanistic reasoning, is required. we wanted digital nations, instead we got digital nationalism one of the most fascinating things that i noticed fairly early in the crypto space starting from around 2014 is just how quickly it started replicating the political patterns of the world at large. i don't mean this just in some broad abstract sense of "people are forming tribes and attacking each other", i mean similarities that are surprisingly deep and specific. first, the story. from 2009 to about 2013, the bitcoin world was a relatively innocent happy place. the community was rapidly growing, prices were rising, and disagreements over block size or long-term direction, while present, were largely academic and took up little attention compared to the shared broader goal of helping bitcoin grow and prosper. but in 2014, the schisms started to arise. transaction volumes on the bitcoin blockchain hit 250 kilobytes per block and kept rising, for the first time raising fears that blockchain usage might actually hit the 1 mb limit before the limit could be increased. non-bitcoin blockchains, up until this point a minor sideshow, suddenly became a major part of the space, with ethereum itself arguably leading the charge. and it was during these events that disagreements that were before politely hidden beneath the surface suddenly blew up. "bitcoin maximalism", the idea that the goal of the crypto space should not be a diverse ecosystem of cryptocurrencies generally but bitcoin and bitcoin alone specifically, grew from a niche curiosity into a prominent and angry movement that dominic williams and i quickly saw for what it is and gave its current name. the small block ideology, arguing that the block size should be increased very slowly or even never increased at all regardless of how high transaction fees go, began to take root. the disagreements within bitcoin would soon turn into an all-out civil war. theymos, the operator of the /r/bitcoin subreddit and several other key public bitcoin discussion spaces, resorted to extreme censorship to impose his (small-block-leaning) views on the community. in response, the big-blockers moved to a new subreddit, /r/btc. some valiantly attempted to defuse tensions with diplomatic conferences including a famous one in hong kong, and a seeming consensus was reached, though one year later the small block side would end up reneging on its part of the deal. by 2017, the big block faction was firmly on its way to defeat, and in august of that year they would secede (or "fork off") to implement their own vision on their own separate continuation of the bitcoin blockchain, which they called "bitcoin cash" (symbol bch). the community split was chaotic, and one can see this in how the channels of communication were split up in the divorce: /r/bitcoin stayed under the control of supporters of bitcoin (btc). /r/btc was controlled by supporters of bitcoin cash (bch). bitcoin.org was controlled by supporters of bitcoin (btc). bitcoin.com on the other hand was controlled by supporters of bitcoin cash (bch). each side claimed themselves to be the true bitcoin. the result looked remarkably similar to one of those civil wars that happens from time to time that results in a country splitting in half, the two halves calling themselves almost identical names that differ only in which subset of the words "democratic", "people's" and "republic" appears on each side. neither side had the ability to destroy the other, and of course there was no higher authority to adjudicate the dispute. major bitcoin forks, as of 2020. does not include bitcoin diamond, bitcoin rhodium, bitcoin private, or any of the other long list of bitcoin forks that i would highly recommend you just ignore completely, except to sell (and perhaps you should sell some of the forks listed above too, eg. bsv is definitely a scam!) around the same time, ethereum had its own chaotic split, in the form of the dao fork, a highly controversial resolution to a theft in which over $50 million was stolen from the first major smart contract application on ethereum. just like in the bitcoin case, there was first a civil war though only lasting four weeks and then a chain split, followed by an online war between the two now-separate chains, ethereum (eth) and ethereum classic (etc). the naming split was as fun as in bitcoin: the ethereum foundation held ethereumproject on twitter but ethereum classic supporters held ethereumproject on github. some on the ethereum side would argue that ethereum classic had very few "real" supporters, and the whole thing was mostly a social attack by bitcoin supporters: either to support the version of ethereum that aligned with their values, or to cause chaos and destroy ethereum outright. i myself believed these claims somewhat at the beginning, though over time i came to realize that they were overhyped. it is true that some bitcoin supporters had certainly tried to shape the outcome in their own image. but to a large extent, as is the case in many conflicts, the "foreign interference" card was simply a psychological defense that many ethereum supporters, myself included, subconsciously used to shield ourselves from the fact that many people within our own community really did have different values. fortunately relations between the two currencies have since improved in part thanks to the excellent diplomatic skills of virgil griffith and ethereum classic developers have even agreed to move to a different github page. civil wars, alliances, blocs, alliances with participants in civil wars, you can all find it in crypto. though fortunately, the conflict is all virtual and online, without the extremely harmful in-person consequences that often come with such things happening in real life. so what can we learn from all this? one important takeaway is this: if phenomena like this happen in contexts as widely different from each other as conflicts between countries, conflicts between religions and relations within and between purely digital cryptocurrencies, then perhaps what we're looking at is the indelible epiphenomena of human nature something much more difficult to resolve than by changing what kinds of groups we organize in. so we should expect situations like this to continue to play out in many contexts over the decades to come. and perhaps it's harder than we thought to separate the good that may come out of this from the bad: those same energies that drive us to fight also drive us to contribute. what motivates us anyway? one of the key intellectual undercurrents of the 2000s era was the recognition of the importance of non-monetary motivations. people are motivated not just by earning as much money as possible in the work and extracting enjoyment from their money in their family lives; even at work we are motivated by social status, honor, altruism, reciprocity, a feeling of contribution, different social conceptions of what is good and valuable, and much more. these differences are very meaningful and measurable. for one example, see this swiss study on compensating differentials for immoral work how much extra do employers have to pay to convince someone to do a job if that job is considered morally unsavory? as we can see, the effects are massive: if a job is widely considered immoral, you need to pay employees almost twice as much for them to be willing to do it. from personal experience, i would even argue that this understates the case: in many cases, top-quality workers would not be willing to work for a company that they think is bad for the world at almost any price. "work" that is difficult to formalize (eg. word-of-mouth marketing) functions similarly: if people think a project is good, they will do it for free, if they do not, they will not do it at all. this is also likely why blockchain projects that raise a lot of money but are unscrupulous, or even just corporate-controlled profit-oriented "vc chains", tend to fail: even a billion dollars of capital cannot compete with a project having a soul. that said, it is possible to be overly idealistic about this fact, in several ways. first of all, while this decentralized, non-market, non-governmental subsidy toward projects that are socially considered to be good is massive, likely amounting to tens of trillions of dollars per year globally, its effect is not infinite. if a developer has a choice between earning $30,000 per year by being "ideologically pure", and making a $30 million ico by sticking a needless token into their project, they will do the latter. second, idealistic motivations are uneven in what they motivate. rick falkvinge's swarmwise played up the possibility of decentralized non-market organization in part by pointing to political activism as a key example. and this is true, political activism does not require getting paid. but longer and more grueling tasks, even something as simple as making good user interfaces, are not so easily intrinsically motivated. and so if you rely on intrinsic motivation too much, you get projects where some tasks are overdone and other tasks are done poorly, or even ignored entirely. and third, perceptions of what people find intrinsically attractive to work on may change, and may even be manipulated. one important conclusion for me from this is the importance of culture (and that oh-so-important word that crypto influencers have unfortunately ruined for me, "narrative"). if a project having a high moral standing is equivalent to that project having twice as much money, or even more, then culture and narrative are extremely powerful forces that command the equivalent of tens of trillions of dollars of value. and this does not even begin to cover the role of such concepts in shaping our perceptions of legitimacy and coordination. and so anything that influences the culture can have a great impact on the world and on people's financial interests, and we're going to see more and more sophisticated efforts from all kinds of actors to do so systematically and deliberately. this is the darker conclusion of the importance of non-monetary social motivations they create the battlefield for the permanent and final frontier of war, the war that is fortunately not usually deadly but unfortunately impossible to create peace treaties for because of how inextricably subjective it is to determine what even counts as a battle: the culture war. big x is here to stay, for all x one of the great debates of the 20th century is that between "big government" and "big business" with various permutations of each: big brother, big banks, big tech, also at times joining the stage. in this environment, the great ideologies were typically defined by trying to abolish the big x that they disliked: communism focusing on corporations, anarcho-capitalism on governments, and so forth. looking back in 2020, one may ask: which of the great ideologies succeeded, and which failed? let us zoom into one specific example: the 1996 declaration of independence of cyberspace: governments of the industrial world, you weary giants of flesh and steel, i come from cyberspace, the new home of mind. on behalf of the future, i ask you of the past to leave us alone. you are not welcome among us. you have no sovereignty where we gather. and the similarly-spirited crypto-anarchist manifesto: computer technology is on the verge of providing the ability for individuals and groups to communicate and interact with each other in a totally anonymous manner. two persons may exchange messages, conduct business, and negotiate electronic contracts without ever knowing the true name, or legal identity, of the other. interactions over networks will be untraceable, via extensive re-routing of encrypted packets and tamper-proof boxes which implement cryptographic protocols with nearly perfect assurance against any tampering. reputations will be of central importance, far more important in dealings than even the credit ratings of today. these developments will alter completely the nature of government regulation, the ability to tax and control economic interactions, the ability to keep information secret, and will even alter the nature of trust and reputation. how have these predictions fared? the answer is interesting: i would say that they succeeded in one part and failed in the other. what succeeded? we have interactions over networks, we have powerful cryptography that is difficult for even state actors to break, we even have powerful cryptocurrency, with smart contract capabilities that the thinkers of the 1990s mostly did not even anticipate, and we're increasingly moving toward anonymized reputation systems with zero knowledge proofs. what failed? well, the government did not go away. and what just proved to be totally unexpected? perhaps the most interesting plot twist is that the two forces are, a few exceptions notwithstanding, by and large not acting like mortal enemies, and there are even many people within governments that are earnestly trying to find ways to be friendly to blockchains and cryptocurrency and new forms of cryptographic trust. what we see in 2020 is this: big government is as powerful as ever, but big business is also as powerful as ever. "big protest mob" is as powerful as ever too, as is big tech, and soon enough perhaps big cryptography. it's a densely populated jungle, with an uneasy peace between many complicated actors. if you define success as the total absence of a category of powerful actor or even a category of activity that you dislike, then you will probably leave the 21st century disappointed. but if you define success more through what happens than through what doesn't happen, and you are okay with imperfect outcomes, there is enough space to make everyone happy. often, the boundary between multiple intersecting worlds is the most interesting place to be. the monkeys get it. prospering in the dense jungle so we have a world where: one-to-one interactions are less important, one-to-many and many-to-many interactions are more important. the environment is much more chaotic, and difficult to model with clean and simple equations. many-to-many interactions particularly follow strange rules that we still do not understand well. the environment is dense, and different categories of powerful actors are forced to live quite closely side by side with each other. in some ways, this is a world that is less convenient for someone like myself. i grew up with a form of economics that is focused on analyzing simpler physical objects and buying and selling, and am now forced to contend with a world where such analysis, while not irrelevant, is significantly less relevant than before. that said, transitions are always challenging. in fact, transitions are particularly challenging for those who think that they are not challenging because they think that the transition merely confirms what they believed all along. if you are still operating today precisely according to a script that was created in 2009, when the great financial crisis was the most recent pivotal event on anyone's mind, then there are almost certainly important things that happened in the last decade that you are missing. an ideology that's finished is an ideology that's dead. it's a world where blockchains and cryptocurrencies are well poised to play an important part, though for reasons much more complex than many people think, and having as much to do with cultural forces as anything financial (one of the more underrated bull cases for cryptocurrency that i have always believed is simply the fact that gold is lame, the younger generations realize that it's lame, and that $9 trillion has to go somewhere). similarly complex forces are what will lead to blockchains and cryptocurrencies being useful. it's easy to say that any application can be done more efficiently with a centralized service, but in practice social coordination problems are very real, and unwillingness to sign onto a system that has even a perception of non-neutrality or ongoing dependence on a third party is real too. and so the centralized and even consortium-based approaches claiming to replace blockchains don't get anywhere, while "dumb and inefficient" public-blockchain-based solutions just keep quietly moving forward and gaining actual adoption. and finally it's a very multidisciplinary world, one that is much harder to break up into layers and analyze each layer separately. you may need to switch from one style of analysis to another style of analysis in mid-sentence. things happen for strange and inscrutable reasons, and there are always surprises. the question that remains is: how do we adapt to it? dark mode toggle el recurso escaso más importante es la legitimidad 2021 mar 23 see all posts un agradecimiento especial a karl floersch, aya miyaguchi, mr silly por sus ideas, comentarios y reseñas y jacobo blandón por traducción. los ecosistemas de cadenas de bloques bitcoin y ethereum gastan más en seguridad de la red objetivo de la minería por prueba de trabajo (pow) que en todo lo demás combinado. la cadena de bloques bitcoin ha pagado un promedio de alrededor de $38 millones al día en recompensas a los mineros desde el comienzo del año, más alrededor de $5 millones en tarifas de transacción diario. la cadena de bloques ethereum ocupa el segundo lugar, con $19.5 millones al día en recompensas en bloque más $18 millones en tarifas de transacción diario. mientras tanto, el presupuesto anual de la fundación ethereum, que paga la investigación, el desarrollo de protocolos, las subvenciones y todo tipo de otros gastos, es de apenas 30 millones de dólares al año. también existen fondos que no provienen de la fundación ethereum, pero a lo sumo son un poco más grandes. es probable que los gastos del ecosistema de bitcoin en i+d sean incluso menores. la i+d del ecosistema bitcoin está financiada en gran parte por empresas (con un total de 250 millones de dólares recaudados hasta ahora según esta página), y este informe sugiere unos 57 empleados; asumiendo salarios bastante altos y muchos desarrolladores pagados que no se cuentan, eso equivale a unos 20 millones de dólares al año. claramente, este patrón de gastos es una mala asignación masiva de recursos. el último 20% de la potencia de hash de la red proporciona mucho menos valor al ecosistema que lo que tendrían esos mismos recursos si se hubieran dedicado a la investigación y al desarrollo del protocolo central. entonces, ¿por qué no ... recortar el presupuesto de pow en un 20% y redirigir los fondos a esas otras cosas en su lugar? la respuesta estándar a este acertijo tiene que ver con conceptos como "teoría de la elección pública" y "vallas de schelling": aunque podríamos identificar fácilmente algunos bienes públicos valiosos a los que redirigir algunos fondos como únicos, haciendo un patrón institucionalizado regular de tales decisiones conllevan riesgos de caos político y captura que a la larga no valen la pena. pero independientemente de las razones por las cuales, nos enfrentamos al hecho interesante que los ecosistemas de bitcoin y ethereum son capaces de convocar miles de millones de dólares de capital, pero tienen restricciones extrañas y difíciles de entender sobre dónde puede ir ese capital. vale la pena comprender la poderosa fuerza social que está creando este efecto. como veremos, también es la misma fuerza social detrás de por qué el ecosistema ethereum es capaz de convocar estos recursos en primer lugar (y que el ecosistema ethereum classic que es casi idéntico tecnológicamente no es capaz). también es una fuerza social que es clave para ayudar a la cadena a recuperarse de un ataque del 51%. y es una fuerza social que subyace a todo tipo de mecanismos extremadamente poderosos mucho más allá del espacio blockchain. por razones que quedarán claras en las próximas secciones, le daré un nombre a esta poderosa fuerza social: legitimidad. las monedas pueden ser propiedad de contratos sociales para comprender mejor la fuerza a la que nos dirigimos, otro ejemplo importante es la saga épica de steem y hive. a principios de 2020, justin sun compró steem-the-company, que no es lo mismo que steem-the-blockchain, pero poseía aproximadamente el 20% del suministro de tokens steem. la comunidad, naturalmente, no confió en justin sun. así que hicieron una votación en cadena para formalizar lo que consideraban un "acuerdo de caballeros", acordando que las monedas de steem-the-company se mantenían en fideicomiso por el bien común de steem-the-blockchain y no deberían usarse para votar. con la ayuda de las monedas en poder de los intercambios, justin sun hizo un contraataque y ganó el control de suficientes delegados para controlar unilateralmente la cadena. la comunidad no vio más opciones dentro del protocolo. entonces, en su lugar, hicieron una bifurcación de steem-the-blockchain, llamada hive, y copiaron todos los saldos de tokens de steem, excepto aquellos, incluido el de justin sun, que participaron en el ataque. y consiguieron muchas aplicaciones a bordo. si no hubieran logrado esto, muchos más usuarios se habrían quedado en steem o se habrían mudado a un proyecto diferente por completo. la lección que podemos aprender de esta situación es la siguiente: steem-the-company nunca "poseyó" las monedas. si lo hubieran hecho, habrían tenido la capacidad práctica de usar, disfrutar y abusar de las monedas de la forma que quisieran. pero en realidad, cuando la empresa trató de disfrutar y abusar de las monedas de una manera que a la comunidad no le gustó, fueron detenidas con éxito. lo que está sucediendo aquí es un patrón similar al que vimos con las recompensas de monedas bitcoin y ethereum aún no emitidas: las monedas finalmente no eran propiedad de una clave criptográfica, sino de algún tipo de contrato social. podemos aplicar el mismo razonamiento a muchas otras estructuras en el espacio blockchain. considere, por ejemplo, la ens root multisig. la root multisig está controlada por siete miembros prominentes de la comunidad de ens y ethereum. pero, ¿qué pasaría si cuatro de ellos se unieran y "actualizaran" el registrador a uno que les transfiera los mejores dominios? dentro del contexto de ens, el-sistema-de-contrato-inteligente, tienen la capacidad completa e indiscutible para hacer esto. pero si realmente intentaran abusar de su capacidad técnica de esta manera, lo que sucedería está claro para cualquiera: serían excluidos de la comunidad, los miembros restantes de la comunidad ens harían un nuevo contrato ens que restaura a los propietarios originales del dominio, y cada aplicación ethereum que usa ens volvería a reescribir su interfaz de usuario para ser usada de nuevo. esto va mucho más allá de las estructuras de contratos inteligentes. ¿por qué elon musk puede vender un nft del tweet de elon musk, pero a jeff bezos le costaría mucho más hacer lo mismo? elon y jeff tienen el mismo nivel de capacidad para capturar el tweet de elon y pegarlo en un dapp nft, entonces, ¿cuál es la diferencia? para cualquiera que tenga una comprensión intuitiva básica de la psicología social humana (o la escena del arte falso), la respuesta es obvia: elon vendiendo el tweet de elon es algo real, y jeff haciendo lo mismo no lo es. una vez más, millones de dólares de valor están siendo controlados y asignados, no por individuos o claves criptográficas, sino por concepciones sociales de legitimidad. y, yendo aún más lejos, la legitimidad gobierna todo tipo de juegos de estatus social, discurso intelectual, lenguaje, derechos de propiedad, sistemas políticos y fronteras nacionales. incluso el consenso de blockchain funciona de la misma manera: la única diferencia entre una bifurcación blanda que es aceptada por la comunidad y un ataque de censura del 51% después del cual la comunidad coordina una bifurcación de recuperación extra-protocolo para eliminar al atacante es la legitimidad. entonces, ¿qué es la legitimidad? ver también: mi anterior publicación sobre la gobernanza de blockchain. para comprender el funcionamiento de la legitimidad, necesitamos profundizar en alguna teoría de juegos.hay muchas situaciones en la vida que exigen un comportamiento coordinado: si actúa de cierta manera solo, es probable que no llegue a ninguna parte (o algo peor), pero si todos actúan juntos, se puede lograr el resultado deseado. un juego de coordinación abstracto. usted se beneficia enormemente de hacer el mismo movimiento que todos los demás. un ejemplo natural es conducir en el lado izquierdo o en el derecho de la carretera: realmente no importa en qué lado de la carretera conduzcan las personas, siempre y cuando conduzcan por el mismo lado. si cambia de bando al mismo tiempo que todos los demás, y la mayoría de la gente prefiere el nuevo arreglo, puede haber un beneficio neto. pero si cambia de lado solo, no importa cuánto prefiera conducir por el otro lado, el resultado neto para usted será bastante negativo. ahora, estamos listos para definir la legitimidad. la legitimidad es un patrón de aceptación de orden superior. un resultado en algún contexto social es legítimo si la gente en ese contexto social acepta ampliamente y juega su papel en la promulgación de ese resultado, y cada persona individual lo hace porque espera que todos los demás hagan lo mismo. la legitimidad es un fenómeno que surge de forma natural en los juegos de coordinación. si usted no está en un juego de coordinación, no hay razón para actuar de acuerdo con sus expectativas de cómo actuarán otras personas, por lo que la legitimidad no es importante. pero, como hemos visto, los juegos de coordinación están en todas partes en la sociedad, por lo que la legitimidad resulta ser bastante importante. en casi cualquier entorno con juegos de coordinación que exista durante bastante tiempo, inevitablemente surgen algunos mecanismos que pueden elegir qué decisión tomar. estos mecanismos están impulsados por una cultura establecida de que todos prestan atención a estos mecanismos y (generalmente) hacen lo que dicen. cada persona razona que debido a que todos los demás siguen estos mecanismos, si hacen algo diferente, solo crearán conflicto y sufrirán, o al menos se quedarán solos en un ecosistema bifurcado y solitario. si un mecanismo tiene la capacidad de tomar estas decisiones con éxito, entonces ese mecanismo tiene legitimidad. un general bizantino reuniendo a sus tropas hacia adelante. el propósito de esto no es solo hacer que los soldados se sientan valientes y emocionados, sino también asegurarles que todos los demás se sienten valientes y emocionados y también cargarán hacia adelante, por lo que un soldado individual no solo se está suicidando al cargar solo hacia adelante. en cualquier contexto en el que haya un juego de coordinación que haya existido durante bastante tiempo, es probable que exista una concepción de legitimidad. y las cadenas de bloques están llenas de juegos de coordinación. ¿qué software de cliente ejecuta? ¿qué registro de nombre de dominio descentralizado solicita qué dirección corresponde a un nombre .eth? ¿qué copia del contrato de uniswap acepta como "el" intercambio de uniswap? incluso los nft son un juego de coordinación. las dos partes más importantes del valor de un nft son (i) el orgullo de tener el nft y la capacidad de mostrar su propiedad, y (ii) la posibilidad de venderlo en el futuro. para ambos componentes, es realmente importante que cualquier nft que compre sea reconocido como legítimo por todos los demás. en todos estos casos, es muy beneficioso tener la misma respuesta que todos los demás, y el mecanismo que determina ese equilibrio tiene mucho poder. teorías de la legitimidad hay muchas formas diferentes en las que puede surgir la legitimidad. en general, la legitimidad surge porque lo que gana legitimidad es psicológicamente atractivo para la mayoría de las personas. pero, por supuesto, las intuiciones psicológicas de las personas pueden ser bastante complejas. es imposible hacer una lista completa de las teorías de la legitimidad, pero podemos comenzar con algunas: legitimidad por fuerza bruta: alguien convence a todos de que son lo suficientemente poderosos como para imponer su voluntad y resistirlos será muy difícil. esto impulsa a la mayoría de las personas a someterse porque cada uno espera que todos los demás también estén demasiado asustados para resistir. legitimidad por continuidad: si algo era legítimo en el momento t, es legítimo por defecto en el momento t + 1. legitimidad por equidad: algo puede volverse legítimo porque satisface una noción intuitiva de justicia. ver también: mi publicación sobre neutralidad creíble, aunque tenga en cuenta que este no es el único tipo de equidad. legitimidad por proceso: si un proceso es legítimo, los resultados de ese proceso ganan legitimidad (por ejemplo, las leyes aprobadas por las democracias a veces se describen de esta manera). legitimidad por desempeño: si los productos de un proceso conducen a resultados que satisfacen a las personas, entonces ese proceso puede ganar legitimidad (por ejemplo, las dictaduras exitosas a veces se describen de esta manera). legitimidad por participación: si las personas participan en la elección de un resultado, es más probable que lo consideren legítimo. esto es similar a la equidad, pero no del todo: se basa en un deseo psicológico de ser coherente con sus acciones anteriores. tenga en cuenta que la legitimidad es un concepto descriptivo; algo puede ser legítimo incluso si usted personalmente piensa que es horrible. dicho esto, si suficientes personas piensan que un resultado es horrible, existe una mayor probabilidad de que suceda algún evento en el futuro que haga que esa legitimidad desaparezca, a menudo al principio de forma gradual y luego repentinamente. la legitimidad es una poderosa tecnología social y deberíamos usarla la situación de la financiación de bienes públicos en los ecosistemas de criptomonedas es bastante mala. hay cientos de miles de millones de dólares de capital circulando, pero los bienes públicos que son clave para la supervivencia continua de ese capital están recibiendo solo decenas de millones de dólares por año de financiación. hay dos formas de responder a este hecho. la primera forma es enorgullecerse de estas limitaciones y de los valientes, aunque no particularmente efectivos, esfuerzos que su comunidad hace para solucionarlos. el autosacrificio personal de los equipos que financian el desarrollo central es, por supuesto, admirable, pero es admirable de la misma manera que eliud kipchoge corriendo un maratón en menos de 2 horas es admirable: es una demostración impresionante de fortaleza humana, pero no es el futuro del transporte. (o, en este caso, financiación de bienes públicos). al igual que tenemos tecnologías mucho mejores para permitir que las personas se muevan 42 km en menos de una hora sin una fortaleza excepcional y años de capacitación, también deberíamos enfocarnos en construir mejores tecnologías sociales para financiar los bienes públicos a las escalas que necesitamos, y como una parte sistémica de nuestra ecología económica y no actos puntuales de iniciativa filantrópica. ahora, volvamos a las criptomonedas. un poder importante de la criptomoneda (y otros activos digitales como nombres de dominio, tierra virtual y nft) está permitiendo a las comunidades reunir grandes cantidades de capital sin que ninguna persona individual tenga que donar personalmente ese capital. sin embargo, este capital está limitado por concepciones de legitimidad: no se puede simplemente asignarlo a un equipo centralizado sin comprometer lo que lo hace valioso. si bien bitcoin y ethereum ya se basan en concepciones de legitimidad para responder a los ataques del 51%, utilizar las concepciones de legitimidad para guiar la financiación de bienes públicos dentro del protocolo es mucho más difícil. pero en la capa de aplicaciones cada vez más rica en la que se crean constantemente nuevos protocolos, tenemos un poco más de flexibilidad en cuanto a dónde podría ir esa financiación. legitimidad en bitshares una de las ideas olvidadas hace mucho tiempo, pero en mi opinión muy innovadoras, del espacio temprano de las criptomonedas fue el modelo de consenso social de bitshares. esencialmente, bitshares se describió a sí misma como una comunidad de personas (titulares de pts y ags) que estaban dispuestas a ayudar a apoyar colectivamente un ecosistema de nuevos proyectos, pero para que un proyecto sea bienvenido en el ecosistema, tendría que asignar el 10% de su token. suministro a los titulares de pts y ags existentes. ahora, por supuesto, cualquiera puede hacer un proyecto que no asigne ninguna moneda a los titulares de pts / ags, o incluso bifurcar un proyecto que sí hizo una asignación y eliminar la asignación. pero, como dice dan larimer: no se puede obligar a nadie a hacer nada, pero en este mercado todo es efecto de red. si a alguien se le ocurre una implementación convincente, entonces puede adoptarla toda la comunidad de pts por el costo de generar un nuevo bloque de génesis. el individuo que decidiera empezar de cero tendría que construir una comunidad completamente nueva alrededor de su sistema. teniendo en cuenta el efecto de red, sospecho que la moneda que honra a protoshares ganará. esta es también una concepción de legitimidad: cualquier proyecto que haga la asignación a los titulares de pts / ags obtendrá la atención y el apoyo de la comunidad (y valdrá la pena que cada miembro de la comunidad individual se interese en el proyecto porque el resto de la comunidad también lo está haciendo), y cualquier proyecto que no haga la asignación no lo hará. ahora bien, esta no es ciertamente una concepción de legitimidad que queremos replicar literalmente (hay poco apetito en la comunidad de ethereum por enriquecer a un pequeño grupo de primeros adoptantes), pero el concepto central se puede adaptar a algo mucho más socialmente valioso. extendiendo el modelo a ethereum los ecosistemas blockchain, incluido ethereum, valoran la libertad y la descentralización. pero, lamentablemente, la ecología de bienes públicos de la mayoría de estas cadenas de bloques sigue estando bastante impulsada por la autoridad y centralización: ya sea ethereum, zcash o cualquier otra cadena de bloques importante, normalmente hay una (o como máximo 2-3) entidades que gastan mucho más que todos. de lo contrario, dar pocas opciones a los equipos independientes que quieran construir bienes públicos. a este modelo de financiación de bienes públicos lo llamo "coordinadores centrales de capital para bienes públicos" (cccp). este estado de cosas no es culpa de las propias organizaciones, que normalmente hacen todo lo posible con valentía para apoyar el ecosistema. más bien, son las reglas del ecosistema las que están siendo injustas con esa organización, porque mantienen a la organización en un estándar injustamente alto. cualquier organización centralizada tendrá inevitablemente puntos ciegos y al menos algunas categorías y equipos cuyo valor no comprenderá; esto no se debe a que cualquiera de los involucrados esté haciendo algo malo, sino a que tal perfección está fuera del alcance de pequeños grupos de humanos. así que hay un gran valor en la creación de un enfoque más diversificado y resistente a los bienes públicos que financian a tomar la presión de una sola organización. afortunadamente, ¡ya tenemos la semilla de esa alternativa! el ecosistema de aplicación de capa en ethereum existe, se está volviendo cada vez más poderosa y ya está mostrando su espíritu público. empresas como gnosis han estado contribuyendo al desarrollo de clientes en ethereum, y varios proyectos ethereum defi han donado cientos de miles de dólares en subvenciones de gitcoin grants. gitcoin grants ya ha alcanzado un alto nivel de legitimidad: su mecanismo de financiación de bienes públicos, la financiación cuadrática, ha demostrado ser credibilidad neutral y eficaz para reflejar las prioridades y valores de la comunidad y tapar los huecos que dejan los mecanismos de financiación existentes. a veces, los principales beneficiarios de contrapartida de gitcoin grants incluso los utilizan como inspiración para las subvenciones por parte de otras entidades otorgantes de subvenciones más centralizadas. la propia fundación ethereum ha desempeñado un papel clave en el apoyo de esta experimentación y diversidad, incubando esfuerzos como gitcoin grants, junto con molochdao y otros, que luego obtienen un apoyo de la comunidad más amplia. podemos fortalecer aún más este incipiente ecosistema de financiamiento de bienes públicos tomando el modelo de bitshares y haciendo una modificación: en lugar de brindar el apoyo comunitario más fuerte a los proyectos que asignaron tokens a una pequeña oligarquía que compró pts o ags en 2013, apoyamos proyectos que aportan una pequeña parte de su tesoro a los bienes públicos que los hacen posibles y al ecosistema del que dependen. y, lo que es más importante, podemos negar estos beneficios a los proyectos que bifurcan un proyecto existente y no devuelven valor al ecosistema en general. hay muchas formas de respaldar los bienes públicos: hacer un compromiso a largo plazo para respaldar el grupo de contrapartida de gitcoin grants, respaldar el desarrollo de clientes en ethereum (también una tarea razonablemente creíble-neutral, ya que existe una definición clara de lo que es un cliente de ethereum), o incluso ejecutando su propio programa de subvenciones cuyo alcance va más allá de ese proyecto de capa de aplicación en particular. la forma más fácil de ponerse de acuerdo sobre lo que cuenta como apoyo suficiente es acordar cuánto, por ejemplo, el 5% del gasto de un proyecto se destinará a respaldar el ecosistema más amplio y otro 1% a bienes públicos que van más allá del espacio blockchain, y dependen de buena fe para elegir adónde irían esos fondos. ¿la comunidad realmente tiene tanta influencia? por supuesto, existen límites para el valor de este tipo de apoyo comunitario. si un proyecto de la competencia (o incluso una bifurcación de un proyecto existente) ofrece a sus usuarios una oferta mucho mejor, entonces los usuarios acudirán en masa, independientemente de cuántas personas les griten que utilicen alguna alternativa que consideren más pro-social. pero estos límites son diferentes en diferentes contextos; a veces, la influencia de la comunidad es débil, pero en otras ocasiones es bastante fuerte. un caso de estudio interesante en este sentido es el caso de tether vs dai. tether tiene muchos escándalos, pero a pesar de esto, los comerciantes usan tether para mantener y mover dólares todo el tiempo. el dai más descentralizado y transparente, a pesar de sus beneficios, no puede quitarle gran parte de la participación de mercado de tether, al menos en lo que respecta a los traders. pero donde dai sobresale son las aplicaciones: augur usa dai, xdai usa dai, pooltogether usa dai, zk.money planea usar dai, y la lista continúa. ¿qué dapps usan usdt? muchas menos. por lo tanto, aunque el poder de los efectos de legitimidad impulsados por la comunidad no es infinito, hay un margen considerable de influencia, suficiente para alentar a los proyectos a dirigir al menos un pequeño porcentaje de sus presupuestos al ecosistema más amplio. incluso hay una razón egoísta para participar en este equilibrio: si fueras el desarrollador de una billetera ethereum, o un autor de un podcast o un boletín informativo, y viste dos proyectos en competencia, uno de los cuales contribuye significativamente a los bienes públicos a nivel del ecosistema, incluyéndote a ti mismo. y uno que no lo hace, ¿por cuál haría todo lo posible para ayudarlos a obtener una mayor participación en el mercado? nft: apoyo a los bienes públicos más allá de ethereum el concepto de apoyar los bienes públicos a través del valor generado "a partir del ether" por concepciones de legitimidad respaldadas públicamente tiene un valor que va mucho más allá del ecosistema ethereum. un desafío y una oportunidad importantes e inmediatos son los nft. los nft tienen una gran posibilidad de ayudar significativamente a muchos tipos de bienes públicos, especialmente de la variedad creativa, a resolver al menos parcialmente sus deficiencias de financiamiento crónicos y sistémicos. de hecho, un primer paso muy admirable. pero también podrían ser una oportunidad perdida: hay poco valor social en ayudar a elon musk a ganar otro millón de dólares vendiendo su tweet cuando, por lo que sabemos, el dinero solo va para él (y, para su crédito, finalmente decidió no vender). si los nft simplemente se convierten en un casino que beneficia en gran medida a las celebridades ya ricas, ese sería un resultado mucho menos interesante. afortunadamente, tenemos la capacidad de ayudar a moldear el resultado. qué nft le resulta atractivo comprar a la gente y cuáles no, es una cuestión de legitimidad: si todos están de acuerdo en que un nft es interesante y otro nft es poco convincente, entonces la gente preferirá comprar el primero, porque tendría ambos más altos valor por los derechos de fanfarronear y el orgullo personal de poseerlo, y porque podría revenderse por más porque todos los demás piensan de la misma manera. si la concepción de la legitimidad de las nft se puede llevar en una buena dirección, existe la oportunidad de establecer un canal sólido de financiación para artistas, organizaciones benéficas y otros. aquí hay dos ideas potenciales: alguna institución (o incluso dao) podría "bendecir" a los nft a cambio de una garantía de que una parte de los ingresos se destina a una causa benéfica, asegurando que varios grupos se beneficien al mismo tiempo. esta bendición podría incluso venir con una categorización oficial: ¿el nft está dedicado al alivio de la pobreza global, la investigación científica, las artes creativas, el periodismo local, el desarrollo de software de código abierto, el empoderamiento de las comunidades marginadas o algo más? podemos trabajar con plataformas de redes sociales para hacer que las nft sean más visibles en los perfiles de las personas, brindando a los compradores una forma de mostrar los valores con los que se comprometieron, no solo con sus palabras, sino con el dinero que tanto les costó ganar. esto podría combinarse con (1) para impulsar a los usuarios hacia las nft que contribuyen a causas sociales valiosas. definitivamente hay más ideas, pero esta es un área que ciertamente merece una coordinación y reflexión más activas. en resumen el concepto de legitimidad (aceptación de orden superior) es muy poderoso. la legitimidad aparece en cualquier contexto donde haya coordinación, y especialmente en internet, la coordinación está en todas partes. hay diferentes formas en las que la legitimidad llega a ser: la fuerza bruta, la continuidad, la equidad, el proceso, el desempeño y la participación están entre las más importantes. *la criptomoneda es poderosa porque nos permite reunir grandes fondos de capital por voluntad económica colectiva, y estos fondos de capital, al principio, no están controlados por ninguna persona. más bien, estos fondos de capital están controlados directamente por conceptos de legitimidad. es demasiado arriesgado comenzar a financiar bienes públicos imprimiendo tokens en la capa base. afortunadamente, sin embargo, ethereum tiene un ecosistema de capas de aplicaciones muy rico, donde tenemos mucha más flexibilidad. esto se debe en parte a que existe la oportunidad no solo de influir en los proyectos existentes, sino también de dar forma a otros nuevos que surgirán en el futuro. los proyectos de la capa de aplicación que apoyan los bienes públicos en la comunidad deben obtener el apoyo de la comunidad, y esto es un gran problema. ¡el ejemplo de dai muestra que este apoyo realmente importa! el ecosistema ethereum se preocupa por el diseño de mecanismos y la innovación en la capa social. ¡los desafíos de financiamiento de bienes públicos del ecosistema ethereum son un excelente lugar para comenzar! pero esto va mucho más allá del propio ethereum. los nft son un ejemplo de una gran cantidad de capital que depende de conceptos de legitimidad. la industria nft podría ser de gran ayuda para los artistas, organizaciones benéficas y otros proveedores de bienes públicos mucho más allá de nuestro propio rincón virtual del mundo, pero este resultado no está predeterminado; depende de la coordinación y el apoyo activos. dark mode toggle exploring fully homomorphic encryption 2020 jul 20 see all posts special thanks to karl floersch and dankrad feist for review fully homomorphic encryption has for a long time been considered one of the holy grails of cryptography. the promise of fully homomorphic encryption (fhe) is powerful: it is a type of encryption that allows a third party to perform computations on encrypted data, and get an encrypted result that they can hand back to whoever has the decryption key for the original data, without the third party being able to decrypt the data or the result themselves. as a simple example, imagine that you have a set of emails, and you want to use a third party spam filter to check whether or not they are spam. the spam filter has a desire for privacy of their algorithm: either the spam filter provider wants to keep their source code closed, or the spam filter depends on a very large database that they do not want to reveal publicly as that would make attacking easier, or both. however, you care about the privacy of your data, and don't want to upload your unencrypted emails to a third party. so here's how you do it: fully homomorphic encryption has many applications, including in the blockchain space. one key example is that can be used to implement privacy-preserving light clients (the light client hands the server an encrypted index i, the server computes and returns data[0] * (i = 0) + data[1] * (i = 1) + ... + data[n] * (i = n), where data[i] is the i'th piece of data in a block or state along with its merkle branch and (i = k) is an expression that returns 1 if i = k and otherwise 0; the light client gets the data it needs and the server learns nothing about what the light client asked). it can also be used for: more efficient stealth address protocols, and more generally scalability solutions to privacy-preserving protocols that today require each user to personally scan the entire blockchain for incoming transactions privacy-preserving data-sharing marketplaces that let users allow some specific computation to be performed on their data while keeping full control of their data for themselves an ingredient in more powerful cryptographic primitives, such as more efficient multi-party computation protocols and perhaps eventually obfuscation and it turns out that fully homomorphic encryption is, conceptually, not that difficult to understand! partially, somewhat, fully homomorphic encryption first, a note on definitions. there are different kinds of homomorphic encryption, some more powerful than others, and they are separated by what kinds of functions one can compute on the encrypted data. partially homomorphic encryption allows evaluating only a very limited set of operations on encrypted data: either just additions (so given encrypt(a) and encrypt(b) you can compute encrypt(a+b)), or just multiplications (given encrypt(a) and encrypt(b) you can compute encrypt(a*b)). somewhat homomorphic encryption allows computing additions as well as a limited number of multiplications (alternatively, polynomials up to a limited degree). that is, if you get encrypt(x1) ... encrypt(xn) (assuming these are "original" encryptions and not already the result of homomorphic computation), you can compute encrypt(p(x1 ... xn)), as long as p(x1 ... xn) is a polynomial with degree < d for some specific degree bound d (d is usually very low, think 5-15). fully homomorphic encryption allows unlimited additions and multiplications. additions and multiplications let you replicate any binary circuit gates (and(x, y) = x*y, or(x, y) = x+y-x*y, xor(x, y) = x+y-2*x*y or just x+y if you only care about even vs odd, not(x) = 1-x...), so this is sufficient to do arbitrary computation on encrypted data. partially homomorphic encryption is fairly easy; eg. rsa has a multiplicative homomorphism: \(enc(x) = x^e\), \(enc(y) = y^e\), so \(enc(x) * enc(y) = (xy)^e = enc(xy)\). elliptic curves can offer similar properties with addition. allowing both addition and multiplication is, it turns out, significantly harder. a simple somewhat-he algorithm here, we will go through a somewhat-homomorphic encryption algorithm (ie. one that supports a limited number of multiplications) that is surprisingly simple. a more complex version of this category of technique was used by craig gentry to create the first-ever fully homomorphic scheme in 2009. more recent efforts have switched to using different schemes based on vectors and matrices, but we will still go through this technique first. we will describe all of these encryption schemes as secret-key schemes; that is, the same key is used to encrypt and decrypt. any secret-key he scheme can be turned into a public key scheme easily: a "public key" is typically just a set of many encryptions of zero, as well as an encryption of one (and possibly more powers of two). to encrypt a value, generate it by adding together the appropriate subset of the non-zero encryptions, and then adding a random subset of the encryptions of zero to "randomize" the ciphertext and make it infeasible to tell what it represents. the secret key here is a large prime, \(p\) (think of \(p\) as having hundreds or even thousands of digits). the scheme can only encrypt 0 or 1, and "addition" becomes xor, ie. 1 + 1 = 0. to encrypt a value \(m\) (which is either 0 or 1), generate a large random value \(r\) (this will typically be even larger than \(p\)) and a smaller random value \(r\) (typically much smaller than \(p\)), and output: \[enc(m) = r * p + r * 2 + m\] to decrypt a ciphertext \(ct\), compute: \[dec(ct) = (ct\ mod\ p)\ mod\ 2\] to add two ciphertexts \(ct_1\) and \(ct_2\), you simply, well, add them: \(ct_1 + ct_2\). and to multiply two ciphertexts, you once again... multiply them: \(ct_1 * ct_2\). we can prove the homomorphic property (that the sum of the encryptions is an encryption of the sum, and likewise for products) as follows. let: \[ct_1 = r_1 * p + r_1 * 2 + m_1\] \[ct_2 = r_2 * p + r_2 * 2 + m_2\] we add: \[ct_1 + ct_2 = r_1 * p + r_2 * p + r_1 * 2 + r_2 * 2 + m_1 + m_2\] which can be rewritten as: \[(r_1 + r_2) * p + (r_1 + r_2) * 2 + (m_1 + m_2)\] which is of the exact same "form" as a ciphertext of \(m_1 + m_2\). if you decrypt it, the first \(mod\ p\) removes the first term, the second \(mod\ 2\) removes the second term, and what's left is \(m_1 + m_2\) (remember that if \(m_1 = 1\) and \(m_2 = 1\) then the 2 will get absorbed into the second term and you'll be left with zero). and so, voila, we have additive homomorphism! now let's check multiplication: \[ct_1 * ct_2 = (r_1 * p + r_1 * 2 + m_1) * (r_2 * p + r_2 * 2 + m_2)\] or: \[(r_1 * r_2 * p + r_1 * 2 + m_1 + r_2 * 2 + m_2) * p + \] \[(r_1 * r_2 * 2 + r_1 * m_2 + r_2 * m_1) * 2 + \] \[(m_1 * m_2)\] this was simply a matter of expanding the product above, and grouping together all the terms that contain \(p\), then all the remaining terms that contain \(2\), and finally the remaining term which is the product of the messages. if you decrypt, then once again the \(mod\ p\) removes the first group, the \(mod\ 2\) removes the second group, and only \(m_1 * m_2\) is left. but there are two problems here: first, the size of the ciphertext itself grows (the length roughly doubles when you multiply), and second, the "noise" (also often called "error") in the smaller \(\* 2\) term also gets quadratically bigger. adding this error into the ciphertexts was necessary because the security of this scheme is based on the approximate gcd problem: had we instead used the "exact gcd problem", breaking the system would be easy: if you just had a set of expressions of the form \(p * r_1 + m_1\), \(p * r_2 + m_2\)..., then you could use the euclidean algorithm to efficiently compute the greatest common divisor \(p\). but if the ciphertexts are only approximate multiples of \(p\) with some "error", then extracting \(p\) quickly becomes impractical, and so the scheme can be secure. unfortunately, the error introduces the inherent limitation that if you multiply the ciphertexts by each other enough times, the error eventually grows big enough that it exceeds \(p\), and at that point the \(mod\ p\) and \(mod\ 2\) steps "interfere" with each other, making the data unextractable. this will be an inherent tradeoff in all of these homomorphic encryption schemes: extracting information from approximate equations "with errors" is much harder than extracting information from exact equations, but any error you add quickly increases as you do computations on encrypted data, bounding the amount of computation that you can do before the error becomes overwhelming. and this is why these schemes are only "somewhat" homomorphic. bootstrapping there are two classes of solution to this problem. first, in many somewhat homomorphic encryption schemes, there are clever tricks to make multiplication only increase the error by a constant factor (eg. 1000x) instead of squaring it. increasing the error by 1000x still sounds by a lot, but keep in mind that if \(p\) (or its equivalent in other schemes) is a 300-digit number, that means that you can multiply numbers by each other 100 times, which is enough to compute a very wide class of computations. second, there is craig gentry's technique of "bootstrapping". suppose that you have a ciphertext \(ct\) that is an encryption of some \(m\) under a key \(p\), that has a lot of error. the idea is that we "refresh" the ciphertext by turning it into a new ciphertext of \(m\) under another key \(p_2\), where this process "clears out" the old error (though it will introduce a fixed amount of new error). the trick is quite clever. the holder of \(p\) and \(p_2\) provides a "bootstrapping key" that consists of an encryption of the bits of \(p\) under the key \(p_2\), as well as the public key for \(p_2\). whoever is doing computations on data encrypted under \(p\) would then take the bits of the ciphertext \(ct\), and individually encrypt these bits under \(p_2\). they would then homomorphically compute the decryption under \(p\) using these ciphertexts, and get out the single bit, which would be \(m\) encrypted under \(p_2\). this is difficult to understand, so we can restate it as follows. the decryption procedure \(dec(ct, p)\) is itself a computation, and so it can itself be implemented as a circuit that takes as input the bits of \(ct\) and the bits of \(p\), and outputs the decrypted bit \(m \in {0, 1}\). if someone has a ciphertext \(ct\) encrypted under \(p\), a public key for \(p_2\), and the bits of \(p\) encrypted under \(p_2\), then they can compute \(dec(ct, p) = m\) "homomorphically", and get out \(m\) encrypted under \(p_2\). notice that the decryption procedure itself washes away the old error; it just outputs 0 or 1. the decryption procedure is itself a circuit, which contains additions or multiplications, so it will introduce new error, but this new error does not depend on the amount of error in the original encryption. (note that we can avoid having a distinct new key \(p_2\) (and if you want to bootstrap multiple times, also a \(p_3\), \(p_4\)...) by just setting \(p_2 = p\). however, this introduces a new assumption, usually called "circular security"; it becomes more difficult to formally prove security if you do this, though many cryptographers think it's fine and circular security poses no significant risk in practice) but.... there is a catch. in the scheme as described above (using circular security or not), the error blows up so quickly that even the decryption circuit of the scheme itself is too much for it. that is, the new \(m\) encrypted under \(p_2\) would already have so much error that it is unreadable. this is because each and gate doubles the bit-length of the error, so a scheme using a \(d\)-bit modulus \(p\) can only handle less than \(log(d)\) multiplications (in series), but decryption requires computing \(mod\ p\) in a circuit made up of these binary logic gates, which requires... more than \(log(d)\) multiplications. craig gentry came up with clever techniques to get around this problem, but they are arguably too complicated to explain; instead, we will skip straight to newer work from 2011 and 2013, that solves this problem in a different way. learning with errors to move further, we will introduce a different type of somewhat-homomorphic encryption introduced by brakerski and vaikuntanathan in 2011, and show how to bootstrap it. here, we will move away from keys and ciphertexts being integers, and instead have keys and ciphertexts be vectors. given a key \(k = {k_1, k_2 .... k_n}\), to encrypt a message \(m\), construct a vector \(c = {c_1, c_2 ... c_n}\) such that the inner product (or "dot product") \(<c, k> = c_1k_1 + c_2k_1 + ... + c_nk_n\), modulo some fixed number \(p\), equals \(m+2e\) where \(m\) is the message (which must be 0 or 1), and \(e\) is a small (much smaller than \(p\)) "error" term. a "public key" that allows encryption but not decryption can be constructed, as before, by making a set of encryptions of 0; an encryptor can randomly combine a subset of these equations and add 1 if the message they are encrypting is 1. to decrypt a ciphertext \(c\) knowing the key \(k\), you would compute \(<c, k>\) modulo \(p\), and see if the result is odd or even (this is the same "mod p mod 2" trick we used earlier). note that here the \(mod\ p\) is typically a "symmetric" mod, that is, it returns a number between \(-\frac{p}{2}\) and \(\frac{p}{2}\) (eg. 137 mod 10 = -3, 212 mod 10 = 2); this allows our error to be positive or negative. additionally, \(p\) does not necessarily have to be prime, though it does need to be odd. key 3 14 15 92 65 ciphertext 2 71 82 81 8 the key and the ciphertext are both vectors, in this example of five elements each. in this example, we set the modulus \(p = 103\). the dot product is 3 * 2 + 14 * 71 + 15 * 82 + 92 * 81 + 65 * 8 = 10202, and \(10202 = 99 * 103 + 5\). 5 itself is of course \(2 * 2 + 1\), so the message is 1. note that in practice, the first element of the key is often set to \(1\); this makes it easier to generate ciphertexts for a particular value (see if you can figure out why). the security of the scheme is based on an assumption known as "learning with errors" (lwe) or, in more jargony but also more understandable terms, the hardness of solving systems of equations with errors. a ciphertext can itself be viewed as an equation: \(k_1c_1 + .... + k_nc_n \approx 0\), where the key \(k_1 ... k_n\) is the unknowns, the ciphertext \(c_1 ... c_n\) is the coefficients, and the equality is only approximate because of both the message (0 or 1) and the error (\(2e\) for some relatively small \(e\)). the lwe assumption ensures that even if you have access to many of these ciphertexts, you cannot recover \(k\). note that in some descriptions of lwe, <c, k> can equal any value, but this value must be provided as part of the ciphertext. this is mathematically equivalent to the <c, k> = m+2e formulation, because you can just add this answer to the end of the ciphertext and add -1 to the end of the key, and get two vectors that when multiplied together just give m+2e. we'll use the formulation that requires <c, k> to be near-zero (ie. just m+2e) because it is simpler to work with. multiplying ciphertexts it is easy to verify that the encryption is additive: if \(<ct_1, k> = 2e_1 + m_1\) and \(<ct_2, k> = 2e_2 + m_2\), then \(<ct_1 + ct_2, k> = 2(e_1 + e_2) + m_1 + m_2\) (the addition here is modulo \(p\)). what is harder is multiplication: unlike with numbers, there is no natural way to multiply two length-n vectors into another length-n vector. the best that we can do is the outer product: a vector containing the products of each possible pair where the first element comes from the first vector and the second element comes from the second vector. that is, \(a \otimes b = a_1b_1 + a_2b_1 + ... + a_nb_1 + a_1b_2 + ... + a_nb_2 + ... + a_nb_n\). we can "multiply ciphertexts" using the convenient mathematical identity \(<a \otimes b, c \otimes d> = <a, c> * <b, d>\). given two ciphertexts \(c_1\) and \(c_2\), we compute the outer product \(c_1 \otimes c_2\). if both \(c_1\) and \(c_2\) were encrypted with \(k\), then \(<c_1, k> = 2e_1 + m_1\) and \(<c_2, k> = 2e_2 + m_2\). the outer product \(c_1 \otimes c_2\) can be viewed as an encryption of \(m_1 * m_2\) under \(k \otimes k\); we can see this by looking what happens when we try to decrypt with \(k \otimes k\): \[<c_1 \otimes c_2, k \otimes k>\] \[= <c_1, k> * <c_2, k>\] \[ = (2e_1 + m_1) * (2e_2 + m_2)\] \[ = 2(e_1m_2 + e_2m_1 + 2e_1e_2) + m_1m_2\] so this outer-product approach works. but there is, as you may have already noticed, a catch: the size of the ciphertext, and the key, grows quadratically. relinearization we solve this with a relinearization procedure. the holder of the private key \(k\) provides, as part of the public key, a "relinearization key", which you can think of as "noisy" encryptions of \(k \otimes k\) under \(k\). the idea is that we provide these encrypted pieces of \(k \otimes k\) to anyone performing the computations, allowing them to compute the equation \(<c_1 \otimes c_2, k \otimes k>\) to "decrypt" the ciphertext, but only in such a way that the output comes back encrypted under \(k\). it's important to understand what we mean here by "noisy encryptions". normally, this encryption scheme only allows encrypting \(m \in \{0,1\}\), and an "encryption of \(m\)" is a vector \(c\) such that \(<c, k> = m+2e\) for some small error \(e\). here, we're "encrypting" arbitrary \(m \in \{0,1, 2....p-1\}\). note that the error means that you can't fully recover \(m\) from \(c\); your answer will be off by some multiple of 2. however, it turns out that, for this specific use case, this is fine. the relinearization key consists of a set of vectors which, when inner-producted (modulo \(p\)) with the key \(k\), give values of the form \(k_i * k_j * 2^d + 2e\) (mod \(p\)), one such vector for every possible triple \((i, j, d)\), where \(i\) and \(j\) are indices in the key and \(d\) is an exponent where \(2^d < p\) (note: if the key has length \(n\), there would be \(n^2 * log(p)\) values in the relinearization key; make sure you understand why before continuing). example assuming p = 15 and k has length 2. formally, enc(x) here means "outputs x+2e if inner-producted with k". now, let us take a step back and look again at our goal. we have a ciphertext which, if decrypted with \(k \otimes k\), gives \(m_1 * m_2\). we want a ciphertext which, if decrypted with \(k\), gives \(m_1 * m_2\). we can do this with the relinearization key. notice that the decryption equation \(<ct_1 \otimes ct_2, k \otimes k>\) is just a big sum of terms of the form \((ct_{1_i} * ct_{2_j}) * k_p * k_q\). and what do we have in our relinearization key? a bunch of elements of the form \(2^d * k_p * k_q\), noisy-encrypted under \(k\), for every possible combination of \(p\) and \(q\)! having all the powers of two in our relinearization key allows us to generate any \((ct_{1_i} * ct_{2_j}) * k_p * k_q\) by just adding up \(\le log(p)\) powers of two (eg. 13 = 8 + 4 + 1) together for each \((p, q)\) pair. for example, if \(ct_1 = [1, 2]\) and \(ct_2 = [3, 4]\), then \(ct_1 \otimes ct_2 = [3, 4, 6, 8]\), and \(enc(<ct_1 \otimes ct_2, k \otimes k>) = enc(3k_1k_1 + 4k_1k_2 + 6k_2k_1 + 8k_2k_2)\) could be computed via: \[enc(k_1 * k_1) + enc(k_1 * k_1 * 2) + enc(k_1 * k_2 * 4) + \] \[enc(k_2 * k_1 * 2) + enc(k_2 * k_1 * 4) + enc(k_2 * k_2 * 8) \] note that each noisy-encryption in the relinearization key has some even error \(2e\), and the equation \(<ct_1 \otimes ct_2, k \otimes k>\) itself has some error: if \(<ct_1, k> = 2e_1 + m_1\) and \(<ct_2 + k> = 2e_2 + m_2\), then \(<ct_1 \otimes ct_2, k \otimes k> =\) \(<ct_1, k> * <ct_2 + k> =\) \(2(2e_1e_2 + e_1m_2 + e_2m_1) + m_1m_2\). but this total error is still (relatively) small (\(2e_1e_2 + e_1m_2 + e_2m_1\) plus \(n^2 * log(p)\) fixed-size errors from the realinearization key), and the error is even, and so the result of this calculation still gives a value which, when inner-producted with \(k\), gives \(m_1 * m_2 + 2e'\) for some "combined error" \(e'\). the broader technique we used here is a common trick in homomorphic encryption: provide pieces of the key encrypted under the key itself (or a different key if you are pedantic about avoiding circular security assumptions), such that someone computing on the data can compute the decryption equation, but only in such a way that the output itself is still encrypted. it was used in bootstrapping above, and it's used here; it's best to make sure you mentally understand what's going on in both cases. this new ciphertext has considerably more error in it: the \(n^2 * log(p)\) different errors from the portions of the relinearization key that we used, plus the \(2(2e_1e_2 + e_1m_2 + e_2m_1)\) from the original outer-product ciphertext. hence, the new ciphertext still does have quadratically larger error than the original ciphertexts, and so we still haven't solved the problem that the error blows up too quickly. to solve this, we move on to another trick... modulus switching here, we need to understand an important algebraic fact. a ciphertext is a vector \(ct\), such that \(<ct, k> = m+2e\), where \(m \in \{0,1\}\). but we can also look at the ciphertext from a different "perspective": consider \(\frac{ct}{2}\) (modulo \(p\)). \(<\frac{ct}{2}, k> = \frac{m}{2} + e\), where \(\frac{m}{2} \in \{0,\frac{p+1}{2}\}\). note that because (modulo \(p\)) \((\frac{p+1}{2})*2 = p+1 = 1\), division by 2 (modulo \(p\)) maps \(1\) to \(\frac{p+1}{2}\); this is a very convenient fact for us. the scheme in this section uses both modular division (ie. multiplying by the modular multiplicative inverse) and regular "rounded down" integer division; make sure you understand how both work and how they are different from each other. that is, the operation of dividing by 2 (modulo \(p\)) converts small even numbers into small numbers, and it converts 1 into \(\frac{p}{2}\) (rounded up). so if we look at \(\frac{ct}{2}\) (modulo \(p\)) instead of \(ct\), decryption involves computing \(<\frac{ct}{2}, k>\) and seeing if it's closer to \(0\) or \(\frac{p}{2}\). this "perspective" is much more robust to certain kinds of errors, where you know the error is small but can't guarantee that it's a multiple of 2. now, here is something we can do to a ciphertext. start: \(<ct, k> = \{0\ or\ 1\} + 2e\ (mod\ p)\) divide \(ct\) by 2 (modulo \(p\)): \(<ct', k> = \{0\ or\ \frac{p}{2}\} + e\ (mod\ p)\) multiply \(ct'\) by \(\frac{q}{p}\) using "regular rounded-down integer division": \(<ct'', k> = \{0\ or\ \frac{q}{2}\} + e' + e_2\ (mod\ q)\) multiply \(ct''\) by 2 (modulo \(q\)): \(<ct''', k> = \{0\ or\ 1\} + 2e' + 2e_2\ (mod\ q)\) step 3 is the crucial one: it converts a ciphertext under modulus \(p\) into a ciphertext under modulus \(q\). the process just involves "scaling down" each element of \(ct'\) by multiplying by \(\frac{q}{p}\) and rounding down, eg. \(floor(56 * \frac{15}{103}) = floor(8.15533..) = 8\). the idea is this: if \(<ct', k> = m*\frac{p}{2} + e\ (mod\ p)\), then we can interpret this as \(<ct', k> = p(z + \frac{m}{2}) + e\) for some integer \(z\). therefore, \(<ct' * \frac{q}{p}, k> = q(z + \frac{m}{2}) + e*\frac{p}{q}\). rounding adds error, but only a little bit (specifically, up to the size of the values in \(k\), and we can make the values in \(k\) small without sacrificing security). therefore, we can say \(<ct' * \frac{q}{p}, k> = m*\frac{q}{2} + e' + e_2\ (mod\ q)\), where \(e' = e * \frac{q}{p}\), and \(e_2\) is a small error from rounding. what have we accomplished? we turned a ciphertext with modulus \(p\) and error \(2e\) into a ciphertext with modulus \(q\) and error \(2(floor(e*\frac{p}{q}) + e_2)\), where the new error is smaller than the original error. let's go through the above with an example. suppose: \(ct\) is just one value, \([5612]\) \(k = [9]\) \(p = 9999\) and \(q = 113\) \(<ct, k> = 5612 * 9 = 50508 = 9999 * 5 + 2 * 256 + 1\), so \(ct\) represents the bit 1, but the error is fairly large (\(e = 256\)). step 2: \(ct' = \frac{ct}{2} = 2806\) (remember this is modular division; if \(ct\) were instead \(5613\), then we would have \(\frac{ct}{2} = 7806\)). checking: \(<ct', k> = 2806 * 9 = 25254 = 9999 * 2.5 + 256.5\) step 3: \(ct'' = floor(2806 * \frac{113}{9999}) = floor(31.7109...) = 31\). checking: \(<ct'', k> = 279 = 113 * 2.5 3.5\) step 4: \(ct''' = 31 * 2 = 62\). checking: \(<ct''', k> = 558 = 113 * 5 2 * 4 + 1\) and so the bit \(1\) is preserved through the transformation. the crazy thing about this procedure is: none of it requires knowing \(k\). now, an astute reader might notice: you reduced the absolute size of the error (from 256 to 2), but the relative size of the error remained unchanged, and even slightly increased: \(\frac{256}{9999} \approx 2.5\%\) but \(\frac{4}{113} \approx 3.5\%\). given that it's the relative error that causes ciphertexts to break, what have we gained here? the answer comes from what happens to error when you multiply ciphertexts. suppose that we start with a ciphertext \(x\) with error 100, and modulus \(p = 10^{16} 1\). we want to repeatedly square \(x\), to compute \((((x^2)^2)^2)^2 = x^{16}\). first, the "normal way": the error blows up too quickly for the computation to be possible. now, let's do a modulus reduction after every multiplication. we assume the modulus reduction is imperfect and increases error by a factor of 10, so a 1000x modulo reduction only reduces error from 10000 to 100 (and not to 10): the key mathematical idea here is that the factor by which error increases in a multiplication depends on the absolute size of the error, and not its relative size, and so if we keep doing modulus reductions to keep the error small, each multiplication only increases the error by a constant factor. and so, with a \(d\) bit modulus (and hence \(\approx 2^d\) room for "error"), we can do \(o(d)\) multiplications! this is enough to bootstrap. another technique: matrices another technique (see gentry, sahai, waters (2013)) for fully homomorphic encryption involves matrices: instead of representing a ciphertext as \(ct\) where \(<ct, k> = 2e + m\), a ciphertext is a matrix, where \(k * ct = k * m + e\) (\(k\), the key, is still a vector). the idea here is that \(k\) is a "secret near-eigenvector" a secret vector which, if you multiply the matrix by it, returns something very close to either zero or the key itself. the fact that addition works is easy: if \(k * ct_1 = m_1 * k + e_1\) and \(k * ct_2 = m_2 * k + e_2\), then \(k * (ct_1 + ct_2) = (m_1 + m_2) * k + (e_1 + e_2)\). the fact that multiplication works is also easy: \(k * ct_1 * ct_2\) \(= (m_1 * k + e_1) * ct_2\) \(= m_1 * k * ct_2 + e_1 * ct_2\) \(= m_1 * m_2 * k + m_1 * e_2 + e_1 * ct_2\) the first term is the "intended term"; the latter two terms are the "error". that said, notice that here error does blow up quadratically (see the \(e_1 * ct_2\) term; the size of the error increases by the size of each ciphertext element, and the ciphertext elements also square in size), and you do need some clever tricks for avoiding this. basically, this involves turning ciphertexts into matrices containing their constituent bits before multiplying, to avoid multiplying by anything higher than 1; if you want to see how this works in detail i recommend looking at my code: https://github.com/vbuterin/research/blob/master/matrix_fhe/matrix_fhe.py#l121 in addition, the code there, and also https://github.com/vbuterin/research/blob/master/tensor_fhe/homomorphic_encryption.py#l186, provides simple examples of useful circuits that you can build out of these binary logical operations; the main example is for adding numbers that are represented as multiple bits, but one can also make circuits for comparison (\(<\), \(>\), \(=\)), multiplication, division, and many other operations. since 2012-13, when these algorithms were created, there have been many optimizations, but they all work on top of these basic frameworks. often, polynomials are used instead of integers; this is called ring lwe. the major challenge is still efficiency: an operation involving a single bit involves multiplying entire matrices or performing an entire relinearization computation, a very high overhead. there are tricks that allow you to perform many bit operations in a single ciphertext operation, and this is actively being worked on and improved. we are quickly getting to the point where many of the applications of homomorphic encryption in privacy-preserving computation are starting to become practical. additionally, research in the more advanced applications of the lattice-based cryptography used in homomorphic encryption is rapidly progressing. so this is a space where some things can already be done today, but we can hopefully look forward to much more becoming possible over the next decade. ethereum’s unexpected future direction | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum’s unexpected future direction posted by vinay gupta on april 1, 2015 research & development ethereum has been preparing what we consider to be the most exciting digital alliance since ibm and microsoft partnered so successfully on the ibm pc. we know that you’ve been anticipating great things from ethereum and we are delighted to present you with this exciting and unexpected news. as you know, mining in ethereum is best done on graphics cards. however, industrial scale buying of expensive graphics cards to mine re-creates the centralized mining cartels which are so much a part of the bitcoin landscape. we would like to be more decentralized than that, and after much consideration and quiet discussion, we would like to announce our durable solution to the mining decentralization problem: valve. valve’s steam is rapidly heading for 100 million users, most of whom have serious graphics hardware at their disposal. valve has the clout to get mining software on to the majority of those machines simply by making it available to users and asking them nicely to download it, in exchange for participation in the network and the chance to make a little credit redeemable for future steam purchases and downloadable content. basically this is ethereum-on-steam: steamium, as we are calling it. this solution solves many of valve’s problems on creating meaningful value for gamers beyond simply having fun too transferable digital property living on the new steamium blockchain will let players buy, sell and trade where appropriate, with smart contracts to prevent cheating, cartelization and illicit sales that could harm the underlying function of the games and the network. of course, steam os and steam machines will likely be steamium-native products, with block chain technology and smart contracts embedded at every level of their function, from verifying and validating software updates through to securing user accounts and allowing users to communicate and transact. our new project intends to go as far as it is possible to go in pushing forwards both gaming and operating systems technology with these new products. one more thing. steamium will be integrated with and released along side of the eagerly-anticipated half life 3.   previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle review of optimism retro funding round 1 2021 nov 16 see all posts special thanks to karl floersch and haonan li for feedback and review, and jinglan wang for discussion. last month, optimism ran their first round of retroactive public goods funding, allocating a total of $1 million to 58 projects to reward the good work that these projects have already done for the optimism and ethereum ecosystems. in addition to being the first major retroactive general-purpose public goods funding experiment, it's also the first experiment in a new kind of governance through badge holders not a very small decision-making board and also not a fully public vote, but instead a quadratic vote among a medium-sized group of 22 participants. the entire process was highly transparent from start to finish: the rules that the badge holders were supposed to follow were enshrined in the badge holder instructions you can see the projects that were nominated in this spreadsheet all discussion between the badge holders happened in publicly viewable forums. in addition to twitter conversation (eg. jeff coleman's thread and also others), all of the explicit structured discussion channels were publicly viewable: the #retroactive-public-goods channel on the optimism discord, and a published zoom call the full results, and the individual badge holder votes that went into the results, can be viewed in this spreadsheet and finally, here are the results in an easy-to-read chart form: much like the gitcoin quadratic funding rounds and the molochdao grants, this is yet another instance of the ethereum ecosystem establishing itself as a key player in the innovative public goods funding mechanism design space. but what can we learn from this experiment? analyzing the results first, let us see if there are any interesting takeaways that can be seen by looking at the results. but what do we compare the results to? the most natural point of comparison is the other major public goods funding experiment that we've had so far: the gitcoin quadratic funding rounds (in this case, round 11). gitcoin round 11 (tech only) optimism retro round 1 probably the most obvious property of the optimism retro results that can be seen without any comparisons is the category of the winners: every major optimism retro winner was a technology project. there was nothing in the badge holder instructions that specified this; non-tech projects (say, the translations at ethereum.cn) were absolutely eligible. and yet, due to some combination of choice of badge holders and subconscious biases, the round seems to have been understood as being tech-oriented. hence, i restricted the gitcoin results in the table above to technology ("dapp tech" + "infra tech") to focus on the remaining differences. some other key remaining differences are: the retro round was low variance: the top-receiving project only got three times more (in fact, exactly three times more) than the 25th, whereas in the gitcoin chart combining the two categories the gap was over 5x, and if you look at dapp tech or infra tech separately the gap is over 15x! i personally blame this on gitcoin using standard quadratic funding (\(reward \approx (\sum_i \sqrt x_i) ^2\)) and the retro round using \(\sum_i \sqrt x_i\) without the square; perhaps the next retro round should just add the square. the retro round winners are more well-known projects: this is actually an intended consequence: the retro round focused on rewarding projects for value already provided, whereas the gitcoin round was open-ended and many contributions were to promising new projects in expectation of future value. the retro round focused more on infrastructure, the gitcoin round more on more user-facing projects: this is of course a generalization, as there are plenty of infrastructure projects in the gitcoin list, but in general applications that are directly user-facing are much more prominent there. a particularly interesting consequence (or cause?) of this is that the gitcoin round more on projects appealing to sub-communities (eg. gamers), whereas the retro round focused more on globally-valuable projects or, less charitably, projects appealing to the one particular sub-community that is ethereum developers. it is my own (admittedly highly subjective) opinion that the retro round winner selection is somewhat higher quality. this is independent of the above three differences; it's more a general impression that the specific projects that were chosen as top recipients on the right were very high quality projects, to a greater extent than top recipients on the left. of course, this could have two causes: (i) a smaller but more skilled number of badge holders ("technocrats") can make better decisions than "the crowd", and (ii) it's easier to judge quality retroactively than ahead of time. and this gets us an interesting question: what if a simple way to summarize much of the above findings is that technocrats are smarter but the crowd is more diverse? could we make badge holders and their outputs more diverse? to better understand the problem, let us zoom in on the one specific example that i already mentioned above: ethereum.cn. this is an excellent chinese ethereum community project (though not the only one! see also ethplanet), which has been providing a lot of resources in chinese for people to learn about ethereum, including translations of many highly technical articles written by ethereum community members and about ethereum originally in english. ethereum.cn webpage. plenty of high quality technical material though they have still not yet gotten the memo that they were supposed to rename "eth2" to "consensus layer". minus ten retroactive reward points for them. what knowledge does a badge holder need to be able to effectively determine whether ethereum.cn is an awesome project, a well-meaning but mediocre project that few chinese people actually visit, or a scam? likely the following: ability to speak and understand chinese being plugged into the chinese community and understanding the social dynamics of that specific project enough understanding of both the tech and of the frame of mind of non-technical readers to judge the site's usefulness for them out of the current, heavily us-focused, badge holders, the number that satisfy these requirements is basically zero. even the two chinese-speaking badge holders are us-based and not close to the chinese ethereum community. so, what happens if we expand the badge holder set? we could add five badge holders from the chinese ethereum community, five from india, five from latin america, five from africa, and five from antarctica to represent the penguins. at the same time we could also diversify among areas of expertise: some technical experts, some community leaders, some people plugged into the ethereum gaming world. hopefully, we can get enough coverage that for each project that's valuable to ethereum, we would have at least 1-5 badge holders who understands enough about that project to be able to intelligently vote on it. but then we see the problem: there would only be 1-5 badge holders able to intelligently vote on it. there are a few families of solutions that i see: tweak the quadratic voting design. in theory, quadratic voting has the unique property that there's very little incentive to vote on projects that you do not understand. any vote you make takes away credits that you could use to vote on projects you understand better. however, the current quadratic voting design has a flaw here: not voting on a project isn't truly a neutral non-vote, it's a vote for the project getting nothing. i don't yet have great ideas for how to do this. a key question is: if zero becomes a truly neutral vote, then how much money does a project get if nobody makes any votes on it? however, this is worth looking into more. split up the voting into categories or sub-committees. badge holders would first vote to sort projects into buckets, and the badge holders within each bucket ("zero knowledge proofs", "games", "india"...) would then make the decisions from there. this could also be done in a more "liquid" way through delegation a badge holder could select some other badge holder to decide their vote on some project, and they would automatically copy their vote. everyone still votes on everything, but facilitate more discussion. the badge holders that do have the needed domain expertise to assess a given project (or a single aspect of some given project) come up with their opinions and write a document or spreadsheet entry to express their reasoning. other badge holders use this information to help make up their minds. once the number of decisions to be made gets even higher, we could even consider ideas like in-protocol random sortition (eg. see this idea to incorporate sortition into quadratic voting) to reduce the number of decisions that each participant needs to make. quadratic sortition has the particularly nice benefit that it naturally leads to large decisions being made by the entire group and small decisions being made by smaller groups. the means-testing debate in the post-round retrospective discussion among the badgeholders, one of the key questions that was brought up is: when choosing which projects to fund, should badge holders take into account whether that project is still in dire need of funding, and de-prioritize projects that are already well-funded through some other means? that is to say, should retroactive rewards be means-tested? in a "regular" grants-funding round, the rationale for answering "yes" is clear: increasing a project's funding from $0 to $100k has a much bigger impact on its ability to do its job than increasing a project's funding from $10m to $10.1m. but optimism retro funding round 1 is not a regular grants-funding round. in retro funding, the objective is not to give people money in expectation of future work that money could help them do. rather, the objective is to reward people for work already done, to change the incentives for anyone working on projects in the future. with this in mind, to what extent should retroactive project funding depend on how much a given project actually needs the funds? the case for means testing suppose that you are a 20-year-old developer, and you are deciding whether to join some well-funded defi project with a fancy token, or to work on some cool open-source fully public good work that will benefit everyone. if you join the well-funded defi project, you will get a $100k salary, and your financial situation will be guaranteed to be very secure. if you work on public-good projects on your own, you will have no income. you have some savings and you could make some money with side gigs, but it will be difficult, and you're not sure if the sacrifice is worth it. now, consider two worlds, world a and world b. first, the similarities: there are ten people exactly like you out there that could get retro rewards, and five of you will. hence, there's a 50% chance you'll get a retro reward. theres's a 30% chance that your work will propel you to moderate fame and you'll be hired by some company with even better terms than the original defi project (or even start your own). now, the differences: world a (means testing): retro rewards are concentrated among the actors that do not find success some other way world b (no means testing): retro rewards are given out independently of whether or not the project finds success in some other way let's look at your chances in each world. event probability (world a) probability (world b) independent success and retroactive reward 0% 15% independent success only 30% 15% retroactive reward only 50% 35% nothing 20% 35% from your point of view as a non-risk-neutral human being, the 15% chance of getting success twice in world b matters much less than the fact that in world b your chances of being left completely in the cold with nothing are nearly double. hence, if we want to encourage people in this hypothetical 20 year old's position to actually contribute, concentrating retro rewards among projects who did not already get rewarded some other way seems prudent. the case against means testing suppose that you are someone who contributes a small amount to many projects, or an investor seed-funding public good projects in anticipation of retroactive rewards. in this case, the share that you would get from any single retroactive reward is small. would you rather have a 10% chance of getting $10,100, or a 10% chance of getting $10,000 and a 10% chance of getting $100? it really doesn't matter. furthermore, your chance of getting rewarded via retroactive funding may well be quite disjoint from your chance of getting rewarded some other way. there are countless stories on the internet of people putting a big part of their lives into a project when that project was non-profit and open-source, seeing that project go for-profit and become successful, and getting absolutely nothing out of it for themselves. in all of these cases, it doesn't really matter whether or not retroactive rewards care on whether or not projects are needy. in fact, it would probably be better for them to just focus on judging quality. means testing has downsides of its own. it would require badge holders to expend effort to determine to what extent a project is well-funded outside the retroactive reward system. it could lead to projects expending effort to hide their wealth and appear scrappy to increase their chance of getting more rewards. subjective evaluations of neediness of recipients could turn into politicized evaluations of moral worthiness of recipients that introduce more controversy into the mechanism. in the extreme, an elaborate tax return system might be required to properly enforce fairness. what do i think? in general, it seems like doing a little bit of prioritizing projects that have not discovered business models has advantages, but we should not do too much of that. projects should be judged by their effect on the world first and foremost. nominations in this round, anyone could nominate projects by submitting them in a google form, and there were only 106 projects nominated. what about the next round, now that people know for sure that they stand a chance at getting thousands of dollars in payout? what about the round a year in the hypothetical future, when fees from millions of daily transactions are paying into the retro funding rounds, and individual projects are getting more money than the entire round is today? some kind of multi-level structure for nominations seems inevitable. there's probably no need to enshrine it directly into the voting rules. instead, we can look at this as one particular way of changing the structure of the discussion: nomination rules filter out the nominations that badge holders need to look at, and anything the badge holders do not look at will get zero votes by default (unless a badge holder really cares to bypass the rules because they have their own reasons to believe that some project is valuable). some possible ideas: badge holder pre-approval: for a proposal to become visible, it must be approved by n badge holders (eg. n=3?). any n badge holders could pre-approve any project; this is an anti-spam speed bump, not a gate-keeping sub-committee. require proposers to provide more information about their proposal, justifying it and reducing the work badge holders need to do to go through it. badge holders would also appoint a separate committee and entrust it with sorting through these proposals and forwarding the ones that follow the rules and pass a basic smell test of not being spam proposals have to specify a category (eg. "zero knowledge proofs", "games", "india"), and badge holders who had declared themselves experts in that category would review those proposals and forward them to a vote only if they chose the right category and pass a basic smell test. proposing requires a deposit of 0.02 eth. if your proposal gets 0 votes (alternatively: if your proposal is explicitly deemed to be "spam"), your deposit is lost. proposing requires a proof-of-humanity id, with a maximum of 3 proposals per human. if your proposals get 0 votes (alternatively: if any of your proposals is explicitly deemed to be "spam"), you can no longer submit proposals for a year (or you have to provide a deposit). conflict of interest rules the first part of the post-round retrospective discussion was taken up by discussion of conflict of interest rules. the badge holder instructions include the following lovely clause:   no self-dealing or conflicts of interest retrodao governance participants should refrain from voting on sending funds to organizations where any portion of those funds is expected to flow to them, their other projects, or anyone they have a close personal or economic relationship with. as far as i can tell, this was honored. badge holders did not try to self-deal, as they were (as far as i can tell) good people, and they knew their reputations were on the line. but there were also some subjective edge cases: wording issues causing confusion. some badge holders wondered about the word "other": could badge holders direct funds to their own primary projects? additionally, the "sending funds to organizations where..." language does not strictly prohibit direct transfers to self. these were arguably simple mistakes in writing this clause; the word "other" should just be removed and "organizations" replaced with "addresses". what if a badge holder is part of a nonprofit that itself gives out grants to other projects? could the badge holder vote for that nonprofit? the badge holder would not benefit, as the funds would 100% pass-through to others, but they could benefit indirectly. what level of connection counts as close connection? ethereum is a tight-knit community and the people qualified to judge the best projects are often at least to some degree friends with the team or personally involved in those projects precisely because they respect those projects. when do those connections step over the line? i don't think there are perfect answers to this; rather, the line will inevitably be gray and can only be discussed and refined over time. the main mechanism-design tweaks that can mitigate it are (i) increasing the number of badge holders, diluting the portion of them that can be insiders in any single project, (ii) reducing the rewards going to projects that only a few badge holders support (my suggestion above to set the reward to \((\sum_i \sqrt x_i)^2\) instead of \(\sum_i \sqrt x_i\) would help here too), and (iii) making sure it's possible for badge holders to counteract clear abuses if they do show up. should badge holder votes be secret ballot? in this round, badge holder votes were completely transparent; anyone can see how each badge holder votes. but transparent voting has a huge downside: it's vulnerable to bribery, including informal bribery of the kind that even good people easily succumb to. badge holders could end up supporting projects in part with the subconscious motivation of winning favor with them. even more realistically, badge holders may be unwilling to make negative votes even when they are justified, because a public negative vote could easily rupture a relationship. secret ballots are the natural alternative. secret ballots are used widely in democratic elections where any citizen (or sometimes resident) can vote, precisely to prevent vote buying and more coercive forms of influencing how people vote. however, in typical elections, votes within executive and legislative bodies are typically public. the usual reasons for this have to do with theories of democratic accountability: voters need to know how their representatives vote so that they can choose their representatives and know that they are not completely lying about their stated values. but there's also a dark side to accountability: elected officials making public votes are accountable to anyone who is trying to bribe them. secret ballots within government bodies do have precedent: the israeli knesset uses secret votes to elect the president and a few other officials the italian parliament has used secret votes in a variety of contexts. in the 19th century, it was considered an important way to protect parliament votes from interference by a monarchy. discussions in us parliaments were less transparent before 1970, and some researchers argue that the switch to more transparency led to more corruption. voting in juries is often secret. sometimes, even the identities of jurors are secret. in general, the conclusion seems to be that secret votes in government bodies have complicated consequences; it's not clear that they should be used everywhere, but it's also not clear that transparency is an absolute good either. in the context of optimism retro funding specifically, the main specific argument i heard against secret voting is that it would make it harder for badge holders to rally and vote against other badge holders making votes that are clearly very wrong or even malicious. today, if a few rogue badge holders start supporting a project that has not provided value and is clearly a cash grab for those badge holders, the other badge holders can see this and make negative votes to counteract this attack. with secret ballots, it's not clear how this could be done. i personally would favor the second round of optimism retro funding using completely secret votes (except perhaps open to a few researchers under conditions of non-disclosure) so we can tell what the material differences are in the outcome. given the current small and tight-knit set of badge holders, dealing with rogue badge hodlers is likely not a primary concern, but in the future it will be; hence, coming up with a secret ballot design that allows counter-voting or some alternative strategy is an important research problem. other ideas for structuring discussion the level of participation among badge holders was very uneven. some (particularly jeff coleman and matt garnett) put a lot of effort into their participation, publicly expressing their detailed reasoning in twitter threads and helping to set up calls for more detailed discussion. others participated in the discussion on discord and still others just voted and did little else. there was a choice made (ok fine, i was the one who suggested it) that the #retroactive-public-goods channel should be readable by all (it's in the optimism discord), but to prevent spam only badge holders should be able to speak. this reduced many people's ability to participate, especially ironically enough my own (i am not a badge holder, and my self-imposed twitter quarantine, which only allows me to tweet links to my own long-form content, prevented me from engaging on twitter). these two factors together meant that there was not that much discussion taking place; certainly less than i had been hoping for. what are some ways to encourage more discussion? some ideas: badge holders could vote in advisors, who cannot vote but can speak in the #retroactive-public-goods channel and other badge-holder-only meetings. badge holders could be required to explain their decisions, eg. writing a post or a paragraph for each project they made votes on. consider compensating badge holders, either through an explicit fixed fee or through a norm that badge holders themselves who made exceptional contributions to discussion are eligible for rewards in future rounds. add more discussion formats. if the number of badge holders increases and there are subgroups with different specialties, there could be more chat rooms and each of them could invite outsiders. another option is to create a dedicated subreddit. it's probably a good idea to start experimenting with more ideas like this. conclusions generally, i think round 1 of optimism retro funding has been a success. many interesting and valuable projects were funded, there was quite a bit of discussion, and all of this despite it only being the first round. there are a number of ideas that could be introduced or experimented with in subsequent rounds: increase the number and diversity of badge holders, while making sure that there is some solution to the problem that only a few badge holders will be experts in any individual project's domain. add some kind of two-layer nomination structure, to lower the decision-making burden that the entire badge holder set is exposed to use secret ballots add more discussion channels, and more ways for non-badge-holders to participate. this could involve reforming how existing channels work, or it could involve adding new channels, or even specialized channels for specific categories of projects. change the reward formula to increase variance, from the current \(\sum_i \sqrt x_i\) to the standard quadratic funding formula of \((\sum_i \sqrt x_i) ^2\). in the long term, if we want retro funding to be a sustainable institution, there is also the question of how new badge holders are to be chosen (and, in cases of malfeasance, how badge holders could be removed). currently, the selection is centralized. in the future, we need some alternative. one possible idea for round 2 is to simply allow existing badge holders to vote in a few new badge holders. in the longer term, to prevent it from being an insular bureaucracy, perhaps one badge holder each round could be chosen by something with more open participation, like a proof-of-humanity vote? in any case, retroactive public goods funding is still an exciting and new experiment in institutional innovation in multiple ways. it's an experiment in non-coin-driven decentralized governance, and it's an experiment in making things happen through retroactive, rather than proactive, incentives. to make the experiment fully work, a lot more innovation will need to happen both in the mechanism itself and in the ecosystem that needs to form around it. when will we see the first retro funding-focused angel investor? whatever ends up happening, i'm looking forward to seeing how this experiment evolves in the rounds to come. cryptographic code obfuscation: decentralized autonomous organizations are about to take a huge leap forward | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search cryptographic code obfuscation: decentralized autonomous organizations are about to take a huge leap forward posted by vitalik buterin on february 8, 2014 research & development there have been a number of very interesting developments in cryptography in the past few years. satoshi’s blockchain notwithstanding, perhaps the first major breakthrough after blinding and zero-knowledge proofs is fully homomorphic encryption, a technology which allows you to upload your data onto a server in an encrypted form so that the server can then perform calculations on it and send you back the results all without having any idea what the data is. in 2013, we saw the beginnings of succinct computational integrity and privacy (scip), a toolkit pioneered by eli ben sasson in israel that lets you cryptographically prove that you carried out some computation and got a certain output. on the more mundane side, we now have sponge functions, an innovation that substantially simplifies the previous mess of hash functions, stream ciphers and pseudorandom number generators into a beautiful, single construction. most recently of all, however, there has been another major development in the cryptographic scene, and one whose applications are potentially very far-reaching both in the cryptocurrency space and for software as a whole: obfuscation. the idea behind obfuscation is an old one, and cryptographers have been trying to crack the problem for years. the problem behind obfuscation is this: is it possible to somehow encrypt a program to produce another program that does the same thing, but which is completely opaque so there is no way to understand what is going on inside? the most obvious use case is proprietary software – if you have a program that incorporates advanced algorithms, and want to let users use the program on specific inputs without being able to reverse-engineer the algorithm, the only way to do such a thing is to obfuscate the code. proprietary software is for obvious reasons unpopular among the tech community, so the idea has not seen a lot of enthusiasm, a problem compounded by the fact that each and every time a company would try to put an obfuscation scheme into practice it would quickly get broken. five years ago, researchers put what might perhaps seem to be a final nail in the coffin: a mathematical proof, using arguments vaguely similar to those used to show the impossibility of the halting problem, that a general purpose obfuscator that converts any program into a “black box” is impossible. at the same time, however, the cryptography community began to follow a different path. understanding that the “black box” ideal of perfect obfuscation will never be achieved, researchers set out to instead aim for a weaker target: indistinguishability obfuscation. the definition of an indistinguishability obfuscator is this: given two programs a and b that compute the same function, if an effective indistinguishability obfuscator o computes two new programs x=o(a) and y=o(b), given x and y there is no (computationally feasible) way to determine which of x and y came from a and which came from b. in theory, this is the best that anyone can do; if there is a better obfuscator, p, then if you put a and p(a) through the indistinguishability obfuscatoro, there would be no way to tell between o(a) and o(p(a)), meaning that the extra step of adding p could not hide any information about the inner workings of the program that o does not. creating such an obfuscator is the problem which many cryptographers have occupied themselves with for the last five years. and in 2013, ucla cryptographer amit sahai, homomorphic encryption pioneer craig gentry and several other researchers figured out how to do it. does the indistinguishability obfuscator actually hide private data inside the program? to see what the answer is, consider the following. suppose your secret password is bobalot_13048, and the sha256 of the password starts with 00b9bbe6345de82f. now, construct two programs. a just outputs 00b9bbe6345de82f, whereas b actually stores bobalot_13048 inside, and when you run it it computes sha256(bobalot_13048) and returns the first 16 hex digits of the output. according to the indistinguishability property, o(a) and o(b) are indistinguishable. if there was some way to extract bobalot_13048 from b, it would therefore be possible to extract bobalot_13048 from a, which essentially implies that you can break sha256 (or by extension any hash function for that matter). by standard assumptions, this is impossible, so therefore the obfuscator must also make it impossible to uncover bobalot_13048 from b. thus, we can be pretty sure that sahai’s obfuscator does actually obfuscate. so what’s the point? in many ways, code obfuscation is one of the holy grails of cryptography. to understand why, consider just how easily nearly every other primitive can be implemented with it. want public key encryption? take any symmetric-key encryption scheme, and construct a decryptor with your secret key built in. obfuscate it, and publish that on the web. you now have a public key. want a signature scheme? public key encryption provides that for you as an easy corollary. want fully homomorphic encryption? construct a program which takes two numbers as an input, decrypts them, adds the results, and encrypts it, and obfuscate the program. do the same for multiplication, send both programs to the server, and the server will swap in your adder and multiplier into its code and perform your computation. however, aside from that, obfuscation is powerful in another key way, and one which has profound consequences particularly in the field of cryptocurrencies and decentralized autonomous organizations: publicly running contracts can now contain private data. on top of second-generation blockchains like ethereum, it will be possible to run so-called “autonomous agents” (or, when the agents primarily serve as a voting system between human actors, “decentralized autonomous organizations”) whose code gets executed entirely on the blockchain, and which have the power to maintain a currency balance and send transactions inside the ethereum system. for example, one might have a contract for a non-profit organization that contains a currency balance, with a rule that the funds can be withdrawn or spent if 67% of the organization’s members agree on the amount and destination to send. unlike bitcoin’s vaguely similar multisig functionality, the rules can be extremely flexible, for example allowing a maximum of 1% per day to be withdrawn with only 33% consent, or making the organization a for-profit company whose shares are tradable and whose shareholders automatically receive dividends. up until now it has been thought that such contracts are fundamentally limited – they can only have an effect inside the ethereum network, and perhaps other systems which deliberately set themselves up to listen to the ethereum network. with obfuscation, however, there are new possibilities. consider the simplest case: an obfuscated ethereum contract can contain a private key to an address inside the bitcoin network, and use that private key to sign bitcoin transactions when the contract’s conditions are met. thus, as long as the ethereum blockchain exists, one can effectively use ethereum as a sort of controller for money that exists inside of bitcoin. from there, however, things only get more interesting. suppose now that you want a decentralized organization to have control of a bank account. with an obfuscated contract, you can have the contract hold the login details to the website of a bank account, and have the contract carry out an entire https session with the bank, logging in and then authorizing certain transfers. you would need some user to act as an intermediary sending packets between the bank and the contract, but this would be a completely trust-free role, like an internet service provider, and anyone could trivially do it and even receive a reward for the task. autonomous agents can now also have social networking accounts, accounts to virtual private servers to carry out more heavy-duty computations than what can be done on a blockchain, and pretty much anything that a normal human or proprietary server can. looking forward thus, we can see that in the next few years decentralized autonomous organizations are potentially going to become much more powerful than they are today. but what are the consequences going to be? in the developed world, the hope is that there will be a massive reduction in the cost of setting up a new business, organization or partnership, and a tool for creating organizations that are much more difficult to corrupt. much of the time, organizations are bound by rules which are really little more than gentlemen’s agreements in practice, and once some of the organization’s members gain a certain measure of power they gain the ability to twist every interpretation in their favor. up until now, the only partial solution was codifying certain rules into contracts and laws – a solution which has its strengths, but which also has its weaknesses, as laws are numerous and very complicated to navigate without the help of a (often very expensive) professional. with daos, there is now also another alternative: making an organization whose organizational bylaws are 100% crystal clear, embedded in mathematical code. of course, there are many things with definitions that are simply too fuzzy to be mathematically defined; in those cases, we will still need some arbitrators, but their role will be reduced to a limited commodity-like function circumscribed by the contract, rather than having potentially full control over everything. in the developing world, however, things will be much more drastic. the developed world has access to a legal system that is at times semi-corrupt, but whose main problems are otherwise simply that it’s too biased toward lawyers and too outdated, bureaucratic and inefficient. the developing world, on the other hand, is plagues by legal systems that are fully corrupt at best, and actively conspiring to pillage their subjects at worst. there, nearly all businesses are gentleman’s agreements, and opportunities for people to betray each other exist at every step. the mathematically encoded organizational bylaws that daos can have are not just an alternative; they may potentially be the first legal system that people have that is actually there to help them. arbitrators can build up their reputations online, as can organizations themselves. ultimately, perhaps on-blockchain voting, like that being pioneered by bitcongress, may even form a basis for new experimental governments. if africa can leapfrog straight from word of mouth communications to mobile phones, why not go from tribal legal systems with the interference of local governments straight to daos? many will of course be concerned that having uncontrollable entities moving money around is dangerous, as there are considerable possibilities for criminal activity with these kinds of powers. to that, however, one can make two simple rebuttals. first, although these decentralized autonomous organizations will be impossible to shut down, they will certainly be very easy to monitor and track every step of the way. it will be possible to detect when one of these entities makes a transaction, it will be easy to see what its balance and relationships are, and it will be possible to glean a lot of information about its organizational structure if voting is done on the blockchain. much like bitcoin, daos are likely far too transparent to be practical for much of the underworld; as fincen director jennifer shasky calvery has recently said, “cash is probably still the best medium for laundering money”. second, ultimately daos cannot do anything normal organizations cannot do; all they are is a set of voting rules for a group of humans or other human-controlled agents to manage ownership of digital assets. even if a dao cannot be shut down, its members certainly can be just as if they were running a plain old normal organization offline. whatever the dominant applications of this new technology turn out to be, one thing is looking more and more certain: cryptography and distributed consensus are about to make the world a whole lot more interesting. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle 為什麼權益證明棒棒的（2020 年十一月） 2020 nov 06 see all posts 感謝chih-cheng liang, hsiao-wei wang和jerry ho翻譯這個文章 權益證明（proof of stake, pos）比工作量證明（proof of work, pow）在區塊鏈安全上更具優勢的關鍵因素有三個。 權益證明在同樣成本下可提供更高的安全性 理解這點最簡單的方式，就是把權益證明和工作量證明擺在一起看，假設每天有 $1 的區塊獎勵，攻擊此網路所需的成本是多少。 基於 gpu 的工作量證明 你可以便宜租到 gpu，所以攻擊網路的成本就等於是租到夠強 gpu 的成本，只要能夠壓過現有礦工 gpu 的算力即可。現有的礦工會花費將近 $1 的成本來賺取 $1 的區塊獎勵（如果他們花費更多，礦工就會因為無利可圖而退出；如果花費比這個數字少，新礦工就會加入來壓縮現有礦工的利潤空間）。因此，攻擊網路只需要暫時花費比 $1/天 多一點點，這還可能只要幾個小時而已。 總攻擊成本： ~$0.26 （假設攻擊 6 小時），且因為攻擊者有機會收到區塊獎勵，所以這個數字可能壓到零 基於 asic 的工作量證明 asics 其實是資本成本：當買進 asic 時你預期大概可以用兩年，因為它會慢慢耗損，或是因為更新更好的硬體出來而過時。如果一個鏈被 51% 攻擊了，社群大概會用更換 pow 演算法來應對，這時，你的 asic 就會失去價值。平均而言，挖礦所費的成本分佈是 1/3 的經常性成本與 2/3 的資本成本（詳見 這裡）。因此，為了每天要賺到 $1 的區塊獎勵，礦工得花 ~$0.33 在電力與維護上，花費 ~$0.67 在 asic 上。假設 asic 可以用 ~2 年，礦工會需要為一單位的 asic 硬體花費 $486.67 。 譯註：$486.67 = 365 天 x 2 年 x $0.67 資本成本 總攻擊成本：$486.67 （asics） + $0.08（電力與維護）= $486.75 譯注：這邊電力與維護成本也是假設攻擊 6 小時 值得注意的是，雖然 asics 提供了很高的安全性，其（相較 gpu ）付出的代價卻更高，讓環境變得過度的中心化。這同時也讓加入 asic 挖礦的門檻變得非常高。 權益證明 proof of stake 權益證明的成本幾乎全是資本成本（抵押的幣）；唯一的營運成本花在運行節點上。猜猜人們願意為每天 $1 的區塊獎勵鎖住多少的資金呢？不像 asic，抵押的幣不會折舊，而且當你不想抵押了，你幾乎馬上就可以取回你的幣。因此，在獎勵相同的情況下，參與者會願意付出比 asic 的情況更多的資本成本。 讓我們假設 ~15% 的報酬率足夠吸引人們抵押（這是 eth2 的期望報酬率）。因此每天 $1 的區塊獎勵會吸引相當於 6.667 年報酬的抵押，可換算成 $2,433 。節點消耗的硬體與電力成本很小，一台千元美金等級的電腦就可以抵押非常大量的幣，而且每月只要花大概 ~$100 左右的電力與網路費就綽綽有餘。我們可以保守的假設上述這些經常性成本為抵押總成本的 10% ，這代表在協議每天發出的區塊獎勵中，需要扣掉 10% 的經常性成本，剩下的 $0.90 才是（攻擊時）需要花費的資本成本。 譯註： 6.667 年 = $1 /（15% 年報酬）； $2,433 = $1 每天 x 365 x 6.667 總攻擊成本： $0.90/天 * 6.667 年 = $2,189 長期來說，這個攻擊成本會變得更高，因為抵押會變得更有效率，人們也因而更能接受較低的報酬率。我個人預期這個數字最終會攀升到 $10,000 左右。 值得注意的是，如此高的安全性帶來的「代價」，只是讓你在抵押時無法任意移動資金而已。人們有可能認知到，這些幣被鎖住之後反而造成了幣的價值攀升，因而造成在社群流通的貨幣總數、像是用於投資生產活動的資金，反而會維持不變。反觀 pow，維持共識的「代價」卻是瘋狂地損耗電力。 我們想要更高的安全性，還是更低的成本？ 要知道的是，在每單位成本安全性增加了 5-20 倍之後，我們有兩種方式可以將其加以利用。一種方式是區塊獎勵維持現狀，單純享受多出的安全性。另一種方式是維持現有程度的安全性，並大量減少區塊獎勵（也就是減少共識機制成本的「浪費」）。 兩種方式都行。我個人喜歡後者，因為我們在下文會看到，跟工作量證明相比，在權益證明中的成功攻擊會造成更少的傷害，而且鏈更容易從攻擊中復原。 權益證明更容易從攻擊中復原 在工作量證明的系統，如果你的鏈遭受 51% 攻擊，你會怎麼做？目前為止，實務上唯一的應對一直是「慢慢等，直到攻擊者覺得無聊了」。但這忽略了一種更危險的攻擊叫做「住到你崩潰(spawn camping attack)」，攻擊者可以對鏈攻擊再攻擊，明確的目標就是要讓鏈無法再使用。 譯注：spawn camping 是一種遊戲術語，指在對方玩家重生點或死亡處埋伏的行為。如此會造成被住、被蹲的玩家一復活就再次陣亡，毫無還手之力。 基於 gpu 的系統完全沒有防禦的辦法，而且持續攻擊的攻擊者可以輕易讓一個鏈永遠毫無用處（實務上來說，該鏈會開始轉移到權益證明或權威證明(proof of authority)）。事實上，在攻擊開始後不久，攻擊者的成本就會變得非常低，而誠實礦工會離開，因為他們沒辦法在攻擊持續之下取得區塊獎勵。 在基於 asic 的系統，社群有辦法應對第一波攻擊，但接下來的攻擊就會變得很容易。社群可以在第一波攻擊之後，硬分叉來更換工作量證明的演算法，讓所有 asic 「變磚」（攻擊者和誠實礦工的 asic 會一起磚掉）。如果攻擊者願意承受第一次 asic 變磚的成本，接下來幾次的情況就和 gpu 的情況一樣（因為還沒有足夠的時間去為新演算法打造與生產 asic），所以在這之後攻擊者可以很便宜地持續對鏈發動攻擊到天荒地老。 譯註：變磚為電子產品俚語，代表損壞後無法使用，像磚頭一樣 在權益證明下，狀況則好上非常多。。針對特定種類的 51% 攻擊（特別指想要推翻已經敲定的區塊），權益證明共識有內建的「砍押金（slashing）」機制，大比例的攻擊者抵押會被自動銷毀（而且不會銷毀到其他人的抵押）。針對其他種類的，更難偵測的攻擊（像是 51% 合謀截斷其他人訊息），社群可以協調一個「少數使用者發起軟分叉 minority user-activated soft fork (uasf)」，可以大量銷毀攻擊者的資金（在以太坊中，可以透過「離線懲罰 inactivity leak 」做到）。如此就不需特別花心力來「搞個硬分叉砍掉非法貨幣」。除了 uasf 需要人為協調要選擇哪個少數區塊，其餘的事情都是自動化的，只要遵照協定規則去執行即可。 譯註：少數區塊 minority block 是小於 51% 抵押總數的驗證者決定出來的區塊。 因此，對鏈的第一次攻擊就會耗損攻擊者幾百萬美元，而且社群可以幾天內馬上站穩腳步。第二次攻擊仍然會花費攻擊者數百萬美元，因為他們被迫要買新的幣來取代自己被燒毀的幣。再攻擊第三次...只會繼續燒更多錢。。局面極為不對稱，而且優勢並不會在攻擊者那邊。 權益證明比 asic 更去中心化 基於 gpu 的工作量證明的去中心化程度還不錯，因為取得 gpu 不會太難。但前面提過，基於 gpu 的挖礦難以滿足「在攻擊之下的安全性」這個準則。另一方面，基於 asic 的挖礦，則需要數百萬美元的資本才能做（而且如果你的 asic 是買來的，多數時候，製造商會佔更多便宜） 如上述，你現在知道要怎麼回答「 pos 只會讓有幣的人繼續以錢滾錢」這個論點了： asic 挖礦一樣會讓已經有幣的人得利 而且賺得比 pos 的時候還多。相比之下，至少權益證明的最低抵押門檻夠低，讓一般人還能有機會接觸。 譯註：就文章完成當下 32 eth @ 440 usd ，最低抵押門檻大概是 40 萬台幣。 進一步說，權益證明更能抵抗審查。gpu 挖礦和 asic 挖礦很容易被偵測，他們需要大量電力消耗、昂貴硬體採購、及大型廠房。另一方面，權益證明可以跑在一台不起眼的筆電上，就算跑在 vpn 上也毫無問題。 工作量證明可能的優勢 我覺得 pow 在下述這兩點上是真的比較佔優勢，縱然這些優勢相當有限。 權益證明更像個「封閉系統」，長期而言財富更加集中 在權益證明的情況下，如果你手上有幣，你可以抵押手上的幣來獲得更多同種類的幣。但，在工作量證明的情況下，就算你沒有幣你還是能賺到，只要你願意投入一些外部資源。因此，或許可以說權益證明會造成長期風險，讓幣的分配變得越來越集中。 我的回應是，在 pos 中，報酬一般而言會很低（所以驗證者的獲利也會低）。在 eth2 ，我們預期驗證者的年報酬會相當於總 eth 供給量的 ~0.5-2%。而且更多驗證者抵押，利率會更低。因此，可能要花個一世紀，整個資產集中程度才會翻倍，而且在這樣的時間跨度之下，其他促進分配的壓力（人們想花他們手上的錢，分配金錢到慈善或他們自己的子孫等等）比較可能會佔上風。 權益證明需要「弱主觀性（weak subjectivity）」而工作量證明不需要 關於「弱主觀性」的觀念可以看這個 原始介紹。本質上，就是節點在第一次上線，或是在離線很長一段時間之後（數個月）再次上線，這個節點必須要透過第三方的資源，才能決定正確的鏈頭在哪。這個第三方可以是他們的朋友、可以是交易所或區塊鏈瀏覽器網站、或是客戶端開發者本身、又或是其他角色。pow 則沒有這樣的要求。 然而，這可能是一個很弱的要求。事實上，使用者本身就已經必須對客戶端開發者、或「社群」有這種程度的信任。最起碼，使用者必須信任某個人（通常是客戶端開發者）來告訴他們協定是什麼，這個協定曾經經歷過什麼樣的更新。這在任何軟體應用都無法避免。因此，這個 pos 所需要增加的額外信任，其實已經算很少了。 但就算事實證明這些風險其實一點都不小，在我看來切換到 pos 系統帶來的好處要高得多，像是系統運作的高效率和從攻擊中回歸正軌的能力。 參考來源：我之前寫權益證明的文章 英：proof of stake faq 英 a proof of stake design philosophy 中 一种权益证明设计哲学 dark mode toggle an approximate introduction to how zk-snarks are possible 2021 jan 26 see all posts special thanks to dankrad feist, karl floersch and hsiao-wei wang for feedback and review. perhaps the most powerful cryptographic technology to come out of the last decade is general-purpose succinct zero knowledge proofs, usually called zk-snarks ("zero knowledge succinct arguments of knowledge"). a zk-snark allows you to generate a proof that some computation has some particular output, in such a way that the proof can be verified extremely quickly even if the underlying computation takes a very long time to run. the "zk" ("zero knowledge") part adds an additional feature: the proof can keep some of the inputs to the computation hidden. for example, you can make a proof for the statement "i know a secret number such that if you take the word ‘cow', add the number to the end, and sha256 hash it 100 million times, the output starts with 0x57d00485aa". the verifier can verify the proof far more quickly than it would take for them to run 100 million hashes themselves, and the proof would also not reveal what the secret number is. in the context of blockchains, this has two very powerful applications: scalability: if a block takes a long time to verify, one person can verify it and generate a proof, and everyone else can just quickly verify the proof instead privacy: you can prove that you have the right to transfer some asset (you received it, and you didn't already transfer it) without revealing the link to which asset you received. this ensures security without unduly leaking information about who is transacting with whom to the public. but zk-snarks are quite complex; indeed, as recently as in 2014-17 they were still frequently called "moon math". the good news is that since then, the protocols have become simpler and our understanding of them has become much better. this post will try to explain how zk-snarks work, in a way that should be understandable to someone with a medium level of understanding of mathematics. note that we will focus on scalability; privacy for these protocols is actually relatively easy once the scalability is there, so we will get back to that topic at the end. why zk-snarks "should" be hard let us take the example that we started with: we have a number (we can encode "cow" followed by the secret input as an integer), we take the sha256 hash of that number, then we do that again another 99,999,999 times, we get the output, and we check what its starting digits are. this is a huge computation. a "succinct" proof is one where both the size of the proof and the time required to verify it grow much more slowly than the computation to be verified. if we want a "succinct" proof, we cannot require the verifier to do some work per round of hashing (because then the verification time would be proportional to the computation). instead, the verifier must somehow check the whole computation without peeking into each individual piece of the computation. one natural technique is random sampling: how about we just have the verifier peek into the computation in 500 different places, check that those parts are correct, and if all 500 checks pass then assume that the rest of the computation must with high probability be fine, too? such a procedure could even be turned into a non-interactive proof using the fiat-shamir heuristic: the prover computes a merkle root of the computation, uses the merkle root to pseudorandomly choose 500 indices, and provides the 500 corresponding merkle branches of the data. the key idea is that the prover does not know which branches they will need to reveal until they have already "committed to" the data. if a malicious prover tries to fudge the data after learning which indices are going to be checked, that would change the merkle root, which would result in a new set of random indices, which would require fudging the data again... trapping the malicious prover in an endless cycle. but unfortunately there is a fatal flaw in naively applying random sampling to spot-check a computation in this way: computation is inherently fragile. if a malicious prover flips one bit somewhere in the middle of a computation, they can make it give a completely different result, and a random sampling verifier would almost never find out. it only takes one deliberately inserted error, that a random check would almost never catch, to make a computation give a completely incorrect result. if tasked with the problem of coming up with a zk-snark protocol, many people would make their way to this point and then get stuck and give up. how can a verifier possibly check every single piece of the computation, without looking at each piece of the computation individually? but it turns out that there is a clever solution. polynomials polynomials are a special class of algebraic expressions of the form: \(x + 5\) \(x^4\) \(x^3 + 3x^2 + 3x + 1\) \(628x^{271} + 318x^{270} + 530x^{269} + ... + 69x + 381\) i.e. they are a sum of any (finite!) number of terms of the form \(c x^k\). there are many things that are fascinating about polynomials. but here we are going to zoom in on a particular one: polynomials are a single mathematical object that can contain an unbounded amount of information (think of them as a list of integers and this is obvious). the fourth example above contained 816 digits of tau, and one can easily imagine a polynomial that contains far more. furthermore, a single equation between polynomials can represent an unbounded number of equations between numbers. for example, consider the equation \(a(x) + b(x) = c(x)\). if this equation is true, then it's also true that: \(a(0) + b(0) = c(0)\) \(a(1) + b(1) = c(1)\) \(a(2) + b(2) = c(2)\) \(a(3) + b(3) = c(3)\) and so on for every possible coordinate. you can even construct polynomials to deliberately represent sets of numbers so you can check many equations all at once. for example, suppose that you wanted to check: 12 + 1 = 13 10 + 8 = 18 15 + 8 = 23 15 + 13 = 28 you can use a procedure called lagrange interpolation to construct polynomials \(a(x)\) that give (12, 10, 15, 15) as outputs at some specific set of coordinates (eg. (0, 1, 2, 3)), \(b(x)\) the outputs (1, 8, 8, 13) on those same coordinates, and so forth. in fact, here are the polynomials: \(a(x) = -2x^3 + \frac{19}{2}x^2 \frac{19}{2}x + 12\) \(b(x) = 2x^3 \frac{19}{2}x^2 + \frac{29}{2}x + 1\) \(c(x) = 5x + 13\) checking the equation \(a(x) + b(x) = c(x)\) with these polynomials checks all four above equations at the same time. comparing a polynomial to itself you can even check relationships between a large number of adjacent evaluations of the same polynomial using a simple polynomial equation. this is slightly more advanced. suppose that you want to check that, for a given polynomial \(f\), \(f(x+2) = f(x) + f(x+1)\) within the integer range \(\{0, 1 ... 98\}\) (so if you also check \(f(0) = f(1) = 1\), then \(f(100)\) would be the 100th fibonacci number). as polynomials, \(f(x+2) f(x+1) f(x)\) would not be exactly zero, as it could give arbitrary answers outside the range \(x = \{0, 1 ... 98\}\). but we can do something clever. in general, there is a rule that if a polynomial \(p\) is zero across some set \(s=\{x_1, x_2 ... x_n\}\) then it can be expressed as \(p(x) = z(x) * h(x)\), where \(z(x) =\) \((x x_1) * (x x_2) * ... * (x x_n)\) and \(h(x)\) is also a polynomial. in other words, any polynomial that equals zero across some set is a (polynomial) multiple of the simplest (lowest-degree) polynomial that equals zero across that same set. why is this the case? it is a nice corollary of polynomial long division: the factor theorem. we know that, when dividing \(p(x)\) by \(z(x)\), we will get a quotient \(q(x)\) and a remainer \(r(x)\) which satisfy \(p(x) = z(x) * q(x) + r(x)\), where the degree of the remainder \(r(x)\) is strictly less than that of \(z(x)\). since we know that \(p\) is zero on all of \(s\), it means that \(r\) has to be zero on all of \(s\) as well. so we can simply compute \(r(x)\) via polynomial interpolation, since it's a polynomial of degree at most \(n-1\) and we know \(n\) values (the zeroes at \(s\)). interpolating a polynomial with all zeroes gives the zero polynomial, thus \(r(x) = 0\) and \(h(x)= q(x)\). going back to our example, if we have a polynomial \(f\) that encodes fibonacci numbers (so \(f(x+2) = f(x) + f(x+1)\) across \(x = \{0, 1 ... 98\}\)), then i can convince you that \(f\) actually satisfies this condition by proving that the polynomial \(p(x) =\) \(f(x+2) f(x+1) f(x)\) is zero over that range, by giving you the quotient: \(h(x) = \frac{f(x+2) f(x+1) f(x)}{z(x)}\) where \(z(x) = (x 0) * (x 1) * ... * (x 98)\). you can calculate \(z(x)\) yourself (ideally you would have it precomputed), check the equation, and if the check passes then \(f(x)\) satisfies the condition! now, step back and notice what we did here. we converted a 100-step-long computation (computing the 100th fibonacci number) into a single equation with polynomials. of course, proving the n'th fibonacci number is not an especially useful task, especially since fibonacci numbers have a closed form. but you can use exactly the same basic technique, just with some extra polynomials and some more complicated equations, to encode arbitrary computations with an arbitrarily large number of steps. now, if only there was a way to verify equations with polynomials that's much faster than checking each coefficient... polynomial commitments and once again, it turns out that there is an answer: polynomial commitments. a polynomial commitment is best viewed as a special way to "hash" a polynomial, where the hash has the additional property that you can check equations between polynomials by checking equations between their hashes. different polynomial commitment schemes have different properties in terms of exactly what kinds of equations you can check. here are some common examples of things you can do with various polynomial commitment schemes (we use \(com(p)\) to mean "the commitment to the polynomial \(p\)"): add them: given \(com(p)\), \(com(q)\) and \(com(r)\) check if \(p + q = r\) multiply them: given \(com(p)\), \(com(q)\) and \(com(r)\) check if \(p * q = r\) evaluate at a point: given \(com(p)\), \(w\), \(z\) and a supplemental proof (or "witness") \(q\), verify that \(p(w) = z\) it's worth noting that these primitives can be constructed from each other. if you can add and multiply, then you can evaluate: to prove that \(p(w) = z\), you can construct \(q(x) = \frac{p(x) z}{x w}\), and the verifier can check if \(q(x) * (x w) + z \stackrel{?}{=} p(x)\). this works because if such a polynomial \(q(x)\) exists, then \(p(x) z = q(x) * (x w)\), which means that \(p(x) z\) equals zero at \(w\) (as \(x w\) equals zero at \(w\)) and so \(p(x)\) equals \(z\) at \(w\). and if you can evaluate, you can do all kinds of checks. this is because there is a mathematical theorem that says, approximately, that if some equation involving some polynomials holds true at a randomly selected coordinate, then it almost certainly holds true for the polynomials as a whole. so if all we have is a mechanism to prove evaluations, we can check eg. our equation \(p(x + 2) p(x + 1) p(x) = z(x) * h(x)\) using an interactive game: as i alluded to earlier, we can make this non-interactive using the fiat-shamir heuristic: the prover can compute r themselves by setting r = hash(com(p), com(h)) (where hash is any cryptographic hash function; it does not need any special properties). the prover cannot "cheat" by picking p and h that "fit" at that particular r but not elsewhere, because they do not know r at the time that they are picking p and h! a quick recap so far zk-snarks are hard because the verifier needs to somehow check millions of steps in a computation, without doing a piece of work to check each individual step directly (as that would take too long). we get around this by encoding the computation into polynomials. a single polynomial can contain an unboundedly large amount of information, and a single polynomial expression (eg. \(p(x+2) p(x+1) p(x) = z(x) * h(x)\)) can "stand in" for an unboundedly large number of equations between numbers. if you can verify the equation with polynomials, you are implicitly verifying all of the number equations (replace \(x\) with any actual x-coordinate) simultaneously. we use a special type of "hash" of a polynomial, called a polynomial commitment, to allow us to actually verify the equation between polynomials in a very short amount of time, even if the underlying polynomials are very large. so, how do these fancy polynomial hashes work? there are three major schemes that are widely used at the moment: bulletproofs, kate and fri. here is a description of kate commitments by dankrad feist: https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html here is a description of bulletproofs by the curve25519-dalek team: https://doc-internal.dalek.rs/bulletproofs/notes/inner_product_proof/index.html, and here is an explanation-in-pictures by myself: https://twitter.com/vitalikbuterin/status/1371844878968176647 here is a description of fri by... myself: ../../../2017/11/22/starks_part_2.html whoa, whoa, take it easy. try to explain one of them simply, without shipping me off to even more scary links to be honest, they're not that simple. there's a reason why all this math did not really take off until 2015 or so. please? in my opinion, the easiest one to understand fully is fri (kate is easier if you're willing to accept elliptic curve pairings as a "black box", but pairings are really complicated, so altogether i find fri simpler). here is how a simplified version of fri works (the real protocol has many tricks and optimizations that are missing here for simplicity). suppose that you have a polynomial \(p\) with degree \(< n\). the commitment to \(p\) is a merkle root of a set of evaluations to \(p\) at some set of pre-selected coordinates (eg. \(\{0, 1 .... 8n-1\}\), though this is not the most efficient choice). now, we need to add something extra to prove that this set of evaluations actually is a degree \(< n\) polynomial. let \(q\) be the polynomial only containing the even coefficients of \(p\), and \(r\) be the polynomial only containing the odd coefficients of \(p\). so if \(p(x) = x^4 + 4x^3 + 6x^2 + 4x + 1\), then \(q(x) = x^2 + 6x + 1\) and \(r(x) = 4x + 4\) (note that the degrees of the coefficients get "collapsed down" to the range \([0...\frac{n}{2})\)). notice that \(p(x) = q(x^2) + x * r(x^2)\) (if this isn't immediately obvious to you, stop and think and look at the example above until it is). we ask the prover to provide merkle roots for \(q(x)\) and \(r(x)\). we then generate a random number \(r\) and ask the prover to provide a "random linear combination" \(s(x) = q(x) + r * r(x)\). we pseudorandomly sample a large set of indices (using the already-provided merkle roots as the seed for the randomness as before), and ask the prover to provide the merkle branches for \(p\), \(q\), \(r\) and \(s\) at these indices. at each of these provided coordinates, we check that: \(p(x)\) actually does equal \(q(x^2) + x * r(x^2)\) \(s(x)\) actually does equal \(q(x) + r * r(x)\) if we do enough checks, then we can be convinced that the "expected" values of \(s(x)\) are different from the "provided" values in at most, say, 1% of cases. notice that \(q\) and \(r\) both have degree \(< \frac{n}{2}\). because \(s\) is a linear combination of \(q\) and \(r\), \(s\) also has degree \(< \frac{n}{2}\). and this works in reverse: if we can prove \(s\) has degree \(< \frac{n}{2}\), then the fact that it's a randomly chosen combination prevents the prover from choosing malicious \(q\) and \(r\) with hidden high-degree coefficients that "cancel out", so \(q\) and \(r\) must both be degree \(< \frac{n}{2}\), and because \(p(x) = q(x^2) + x * r(x^2)\), we know that \(p\) must have degree \(< n\). from here, we simply repeat the game with \(s\), progressively "reducing" the polynomial we care about to a lower and lower degree, until it's at a sufficiently low degree that we can check it directly. as in the previous examples, "bob" here is an abstraction, useful for cryptographers to mentally reason about the protocol. in reality, alice is generating the entire proof herself, and to prevent her from cheating we use fiat-shamir: we choose each randomly samples coordinate or r value based on the hash of the data generated in the proof up until that point. a full "fri commitment" to \(p\) (in this simplified protocol) would consist of: the merkle root of evaluations of \(p\) the merkle roots of evaluations of \(q\), \(r\), \(s_1\) the randomly selected branches of \(p\), \(q\), \(r\), \(s_1\) to check \(s_1\) is correctly "reduced from" \(p\) the merkle roots and randomly selected branches just as in steps (2) and (3) for successively lower-degree reductions \(s_2\) reduced from \(s_1\), \(s_3\) reduced from \(s_2\), all the way down to a low-degree \(s_k\) (this gets repeated \(\approx log_2(n)\) times in total) the full merkle tree of the evaluations of \(s_k\) (so we can check it directly) each step in the process can introduce a bit of "error", but if you add enough checks, then the total error will be low enough that you can prove that \(p(x)\) equals a degree \(< n\) polynomial in at least, say, 80% of positions. and this is sufficient for our use cases. if you want to cheat in a zk-snark, you would need to make a polynomial commitment for a fractional expression (eg. to "prove" the false claim that \(x^2 + 2x + 3\) evaluated at \(4\) equals \(5\), you would need to provide a polynomial commitment for \(\frac{x^2 + 2x + 3 5}{x 4} = x + 6 + \frac{22}{x 4}\)). the set of evaluations for such a fractional expression would differ from the evaluations for any real degree \(< n\) polynomial in so many positions that any attempt to make a fri commitment to them would fail at some step. also, you can check carefully that the total number and size of the objects in the fri commitment is logarithmic in the degree, so for large polynomials, the commitment really is much smaller than the polynomial itself. to check equations between different polynomial commitments of this type (eg. check \(a(x) + b(x) = c(x)\) given fri commitments to \(a\), \(b\) and \(c\)), simply randomly select many indices, ask the prover for merkle branches at each of those indices for each polynomial, and verify that the equation actually holds true at each of those positions. the above description is a highly inefficient protocol; there is a whole host of algebraic tricks that can increase its efficiency by a factor of something like a hundred, and you need these tricks if you want a protocol that is actually viable for, say, use inside a blockchain transaction. in particular, for example, \(q\) and \(r\) are not actually necessary, because if you choose your evaluation points very cleverly, you can reconstruct the evaluations of \(q\) and \(r\) that you need directly from evaluations of \(p\). but the above description should be enough to convince you that a polynomial commitment is fundamentally possible. finite fields in the descriptions above, there was a hidden assumption: that each individual "evaluation" of a polynomial was small. but when we are dealing with polynomials that are big, this is clearly not true. if we take our example from above, \(628x^{271} + 318x^{270} + 530x^{269} + ... + 69x + 381\), that encodes 816 digits of tau, and evaluate it at \(x=1000\), you get.... an 816-digit number containing all of those digits of tau. and so there is one more thing that we need to add. in a real implementation, all of the arithmetic that we are doing here would not be done using "regular" arithmetic over real numbers. instead, it would be done using modular arithmetic. we redefine all of our arithmetic operations as follows. we pick some prime "modulus" p. the % operator means "take the remainder of": \(15\ \%\ 7 = 1\), \(53\ \%\ 10 = 3\), etc (note that the answer is always non-negative, so for example \(-1\ \%\ 10 = 9\)). we redefine \(x + y \rightarrow (x + y)\) % \(p\) \(x * y \rightarrow (x * y)\) % \(p\) \(x^y \rightarrow (x^y)\) % \(p\) \(x y \rightarrow (x y)\) % \(p\) \(x / y \rightarrow (x * y ^{p-2})\) % \(p\) the above rules are all self-consistent. for example, if \(p = 7\), then: \(5 + 3 = 1\) (as \(8\) % \(7 = 1\)) \(1 3 = 5\) (as \(-2\) % \(7 = 5\)) \(2 \cdot 5 = 3\) \(3 / 5 = 2\) (as (\(3 \cdot 5^5\)) % \(7 = 9375\) % \(7 = 2\)) more complex identities such as the distributive law also hold: \((2 + 4) \cdot 3\) and \(2 \cdot 3 + 4 \cdot 3\) both evaluate to \(4\). even formulas like \((a^2 b^2)\) = \((a b) \cdot (a + b)\) are still true in this new kind of arithmetic. division is the hardest part; we can't use regular division because we want the values to always remain integers, and regular division often gives non-integer results (as in the case of \(3/5\)). we get around this problem using fermat's little theorem, which states that for any nonzero \(x < p\), it holds that \(x^{p-1}\) % \(p = 1\). this implies that \(x^{p-2}\) gives a number which, if multiplied by \(x\) one more time, gives \(1\), and so we can say that \(x^{p-2}\) (which is an integer) equals \(\frac{1}{x}\). a somewhat more complicated but faster way to evaluate this modular division operator is the extended euclidean algorithm, implemented in python here. because of how the numbers "wrap around", modular arithmetic is sometimes called "clock math" with modular math we've created an entirely new system of arithmetic, and it's self-consistent in all the same ways traditional arithmetic is self-consistent. hence, we can talk about all of the same kinds of structures over this field, including polynomials, that we talk about in "regular math". cryptographers love working in modular math (or, more generally, "finite fields") because there is a bound on the size of a number that can arise as a result of any modular math calculation no matter what you do, the values will not "escape" the set \(\{0, 1, 2 ... p-1\}\). even evaluating a degree-1-million polynomial in a finite field will never give an answer outside that set. what's a slightly more useful example of a computation being converted into a set of polynomial equations? let's say we want to prove that, for some polynomial \(p\), \(0 \le p(n) < 2^{64}\), without revealing the exact value of \(p(n)\). this is a common use case in blockchain transactions, where you want to prove that a transaction leaves a balance non-negative without revealing what that balance is. we can construct a proof for this with the following polynomial equations (assuming for simplicity \(n = 64\)): \(p(0) = 0\) \(p(x+1) = p(x) * 2 + r(x)\) across the range \(\{0...63\}\) \(r(x) \in \{0,1\}\) across the range \(\{0...63\}\) the latter two statements can be restated as "pure" polynomial equations as follows (in this context \(z(x) = (x 0) * (x 1) * ... * (x 63)\)): \(p(x+1) p(x) * 2 r(x) = z(x) * h_1(x)\) \(r(x) * (1 r(x)) = z(x) * h_2(x)\) (notice the clever trick: \(y * (1-y) = 0\) if and only if \(y \in \{0, 1\}\)) the idea is that successive evaluations of \(p(i)\) build up the number bit-by-bit: if \(p(4) = 13\), then the sequence of evaluations going up to that point would be: \(\{0, 1, 3, 6, 13\}\). in binary, 1 is 1, 3 is 11, 6 is 110, 13 is 1101; notice how \(p(x+1) = p(x) * 2 + r(x)\) keeps adding one bit to the end as long as \(r(x)\) is zero or one. any number within the range \(0 \le x < 2^{64}\) can be built up over 64 steps in this way, any number outside that range cannot. privacy but there is a problem: how do we know that the commitments to \(p(x)\) and \(r(x)\) don't "leak" information that allows us to uncover the exact value of \(p(64)\), which we are trying to keep hidden? there is some good news: these proofs are small proofs that can make statements about a large amount of data and computation. so in general, the proof will very often simply not be big enough to leak more than a little bit of information. but can we go from "only a little bit" to "zero"? fortunately, we can. here, one fairly general trick is to add some "fudge factors" into the polynomials. when we choose \(p\), add a small multiple of \(z(x)\) into the polynomial (that is, set \(p'(x) = p(x) + z(x) * e(x)\) for some random \(e(x)\)). this does not affect the correctness of the statement (in fact, \(p'\) evaluates to the same values as \(p\) on the coordinates that "the computation is happening in", so it's still a valid transcript), but it can add enough extra "noise" into the commitments to make any remaining information unrecoverable. additionally, in the case of fri, it's important to not sample random points that are within the domain that computation is happening in (in this case \(\{0...64\}\)). can we have one more recap, please?? the three most prominent types of polynomial commitments are fri, kate and bulletproofs. kate is the simplest conceptually but depends on the really complicated "black box" of elliptic curve pairings. fri is cool because it relies only on hashes; it works by successively reducing a polynomial to a lower and lower-degree polynomial and doing random sample checks with merkle branches to prove equivalence at each step. to prevent the size of individual numbers from blowing up, instead of doing arithmetic and polynomials over the integers, we do everything over a finite field (usually integers modulo some prime p) polynomial commitments lend themselves naturally to privacy preservation because the proof is already much smaller than the polynomial, so a polynomial commitment can't reveal more than a little bit of the information in the polynomial anyway. but we can add some randomness to the polynomials we're committing to to reduce the information revealed from "a little bit" to "zero". what research questions are still being worked on? optimizing fri: there are already quite a few optimizations involving carefully selected evaluation domains, "deep-fri", and a whole host of other tricks to make fri more efficient. starkware and others are working on this. better ways to encode computation into polynomials: figuring out the most efficient way to encode complicated computations involving hash functions, memory access and other features into polynomial equations is still a challenge. there has been great progress on this (eg. see plookup), but we still need more, especially if we want to encode general-purpose virtual machine execution into polynomials. incrementally verifiable computation: it would be nice to be able to efficiently keep "extending" a proof while a computation continues. this is valuable in the "single-prover" case, but also in the "multi-prover" case, particularly a blockchain where a different participant creates each block. see halo for some recent work on this. i wanna learn more! my materials starks: part 1, part 2, part 3 specific protocols for encoding computation into polynomials: plonk some key mathematical optimizations i didn't talk about here: fast fourier transforms other people's materials starkware's online course dankrad feist on kate commitments bulletproofs dark mode toggle daos are not corporations: where decentralization in autonomous organizations matters 2022 sep 20 see all posts special thanks to karl floersch and tina zhen for feedback and review on earlier versions of this article. recently, there has been a lot of discourse around the idea that highly decentralized daos do not work, and dao governance should start to more closely resemble that of traditional corporations in order to remain competitive. the argument is always similar: highly decentralized governance is inefficient, and traditional corporate governance structures with boards, ceos and the like evolved over hundreds of years to optimize for the goal of making good decisions and delivering value to shareholders in a changing world. dao idealists are naive to assume that egalitarian ideals of decentralization can outperform this, when attempts to do this in the traditional corporate sector have had marginal success at best. this post will argue why this position is often wrong, and offer a different and more detailed perspective about where different kinds of decentralization are important. in particular, i will focus on three types of situations where decentralization is important: decentralization for making better decisions in concave environments, where pluralism and even naive forms of compromise are on average likely to outperform the kinds of coherency and focus that come from centralization. decentralization for censorship resistance: applications that need to continue functioning while resisting attacks from powerful external actors. decentralization as credible fairness: applications where daos are taking on nation-state-like functions like basic infrastructure provision, and so traits like predictability, robustness and neutrality are valued above efficiency. centralization is convex, decentralization is concave see the original post: ../../../2020/11/08/concave.html one way to categorize decisions that need to be made is to look at whether they are convex or concave. in a choice between a and b, we would first look not at the question of a vs b itself, but instead at a higher-order question: would you rather take a compromise between a and b or a coin flip? in expected utility terms, we can express this distinction using a graph: if a decision is concave, we would prefer a compromise, and if it's convex, we would prefer a coin flip. often, we can answer the higher-order question of whether a compromise or a coin flip is better much more easily than we can answer the first-order question of a vs b itself. examples of convex decisions include: pandemic response: a 100% travel ban may work at keeping a virus out, a 0% travel ban won't stop viruses but at least doesn't inconvenience people, but a 50% or 90% travel ban is the worst of both worlds. military strategy: attacking on front a may make sense, attacking on front b may make sense, but splitting your army in half and attacking at both just means the enemy can easily deal with the two halves one by one technology choices in crypto protocols: using technology a may make sense, using technology b may make sense, but some hybrid between the two often just leads to needless complexity and even adds risks of the two interfering with each other. examples of concave decisions include: judicial decisions: an average between two independently chosen judgements is probably more likely to be fair, and less likely to be completely ridiculous, than a random choice of one of the two judgements. public goods funding: usually, giving $x to each of two promising projects is more effective than giving $2x to one and nothing to the other. having any money at all gives a much bigger boost to a project's ability to achieve its mission than going from $x to $2x does. tax rates: because of quadratic deadweight loss mechanics, a tax rate of x% is often only a quarter as harmful as a tax rate of 2x%, and at the same time more than half as good at raising revenue. hence, moderate taxes are better than a coin flip between low/no taxes and high taxes. when decisions are convex, decentralizing the process of making that decision can easily lead to confusion and low-quality compromises. when decisions are concave, on the other hand, relying on the wisdom of the crowds can give better answers. in these cases, dao-like structures with large amounts of diverse input going into decision-making can make a lot of sense. and indeed, people who see the world as a more concave place in general are more likely to see a need for decentralization in a wider variety of contexts. should vitadao and ukraine dao be daos? many of the more recent daos differ from earlier daos, like makerdao, in that whereas the earlier daos are organized around providing infrastructure, the newer daos are organized around performing various tasks around a particular theme. vitadao is a dao funding early-stage longevity research, and ukrainedao is a dao organizing and funding efforts related to helping ukrainian victims of war and supporting the ukrainian defense effort. does it make sense for these to be daos? this is a nuanced question, and we can get a view of one possible answer by understanding the internal workings of ukrainedao itself. typical daos tend to "decentralize" by gathering large amounts of capital into a single pool and using token-holder voting to fund each allocation. ukrainedao, on the other hand, works by splitting its functions up into many pods, where each pod works as independently as possible. a top layer of governance can create new pods (in principle, governance can also fund pods, though so far funding has only gone to external ukraine-related organizations), but once a pod is made and endowed with resources, it functions largely on its own. internally, individual pods do have leaders and function in a more centralized way, though they still try to respect an ethos of personal autonomy. one natural question that one might ask is: isn't this kind of "dao" just rebranding the traditional concept of multi-layer hierarchy? i would say this depends on the implementation: it's certainly possible to take this template and turn it into something that feels authoritarian in the same way stereotypical large corporations do, but it's also possible to use the template in a very different way. two things that can help ensure that an organization built this way will actually turn out to be meaningfully decentralized include: a truly high level of autonomy for pods, where the pods accept resources from the core and are occasionally checked for alignment and competence if they want to keep getting those resources, but otherwise act entirely on their own and don't "take orders" from the core. highly decentralized and diverse core governance. this does not require a "governance token", but it does require broader and more diverse participation in the core. normally, broad and diverse participation is a large tax on efficiency. but if (1) is satisfied, so pods are highly autonomous and the core needs to make fewer decisions, the effects of top-level governance being less efficient become smaller. now, how does this fit into the "convex vs concave" framework? here, the answer is roughly as follows: the (more decentralized) top level is concave, the (more centralized within each pod) bottom level is convex. giving a pod $x is generally better than a coin flip between giving it $0 and giving it $2x, and there isn't a large loss from having compromises or "inconsistent" philosophies guiding different decisions. but within each individual pod, having a clear opinionated perspective guiding decisions and being able to insist on many choices that have synergies with each other is much more important. decentralization and censorship resistance the most often publicly cited reason for decentralization in crypto is censorship resistance: a dao or protocol needs to be able to function and defend itself despite external attack, including from large corporate or even state actors. this has already been publicly talked about at length, and so deserves less elaboration, but there are still some important nuances. two of the most successful censorship-resistant services that large numbers of people use today are the pirate bay and sci-hub. the pirate bay is a hybrid system: it's a search engine for bittorrent, which is a highly decentralized network, but the search engine itself is centralized. it has a small core team that is dedicated to keeping it running, and it defends itself with the mole's strategy in whack-a-mole: when the hammer comes down, move out of the way and re-appear somewhere else. the pirate bay and sci-hub have both frequently changed domain names, relied on arbitrage between different jurisdictions, and used all kinds of other techniques. this strategy is centralized, but it has allowed them both to be successful both at defense and at product-improvement agility. daos do not act like the pirate bay and sci-hub; daos act like bittorrent. and there is a reason why bittorrent does need to be decentralized: it requires not just censorship resistance, but also long-term investment and reliability. if bittorrent got shut down once a year and required all its seeders and users to switch to a new provider, the network would quickly degrade in quality. censorship resistance-demanding daos should also be in the same category: they should be providing a service that isn't just evading permanent censorship, but also evading mere instability and disruption. makerdao (and the reflexer dao which manages rai) are excellent examples of this. a dao running a decentralized search engine probably does not: you can just build a regular search engine and use sci-hub-style techniques to ensure its survival. decentralization as credible fairness sometimes, daos' primary concern is not a need to resist nation states, but rather a need to take on some of the functions of nation states. this often involves tasks that can be described as "maintaining basic infrastructure". because governments have less ability to oversee daos, daos need to be structured to take on a greater ability to oversee themselves. and this requires decentralization. of course, it's not actually possible to come anywhere close to eliminating hierarchy and inequality of information and decision-making power in its entirety etc etc etc, but what if we can get even 30% of the way there? consider three motivating examples: algorithmic stablecoins, the kleros court, and the optimism retroactive funding mechanism. an algorithmic stablecoin dao is a system that uses on-chain financial contracts to create a crypto-asset whose price tracks some stable index, often but not necessarily the us dollar. kleros is a "decentralized court": a dao whose function is to give rulings on arbitration questions such as "is this github commit an acceptable submission to this on-chain bounty?" optimism's retroactive funding mechanism is a component of the optimism dao which retroactively rewards projects that have provided value to the ethereum and optimism ecosystems. in all three cases, there is a need to make subjective judgements, which cannot be done automatically through a piece of on-chain code. in the first case, the goal is simply to get reasonably accurate measurements of some price index. if the stablecoin tracks the us dollar, then you just need the eth/usd price. if hyperinflation or some other reason to abandon the us dollar arises, the stablecoin dao might need to manage a trustworthy on-chain cpi calculation. kleros is all about making unavoidably subjective judgements on any arbitrary question that is submitted to it, including whether or not submitted questions should be rejected for being "unethical". optimism's retroactive funding is tasked with one of the most open-ended subjective questions at all: what projects have done work that is the most useful to the ethereum and optimism ecosystems? all three cases have an unavoidable need for "governance", and pretty robust governance too. in all cases, governance being attackable, from the outside or the inside, can easily lead to very big problems. finally, the governance doesn't just need to be robust, it needs to credibly convince a large and untrusting public that it is robust. the algorithmic stablecoin's achilles heel: the oracle algorithmic stablecoins depend on oracles. in order for an on-chain smart contract to know whether to target the value of dai to 0.005 eth or 0.0005 eth, it needs some mechanism to learn the (external-to-the-chain) piece of information of what the eth/usd price is. and in fact, this "oracle" is the primary place at which an algorithmic stablecoin can be attacked. this leads to a security conundrum: an algorithmic stablecoin cannot safely hold more collateral, and therefore cannot issue more units, than the market cap of its speculative token (eg. mkr, flx...), because if it does, then it becomes profitable to buy up half the speculative token supply, use those tokens to control the oracle, and steal funds from users by feeding bad oracle values and liquidating them. here is a possible alternative design for a stablecoin oracle: add a layer of indirection. quoting the ethresear.ch post: we set up a contract where there are 13 "providers"; the answer to a query is the median of the answer returned by these providers. every week, there is a vote, where the oracle token holders can replace one of the providers ... the security model is simple: if you trust the voting mechanism, you can trust the oracle output, unless 7 providers get corrupted at the same time. if you trust the current set of oracle providers, you can trust the output for at least the next six weeks, even if you completely do not trust the voting mechanism. hence, if the voting mechanism gets corrupted, there will be able time for participants in any applications that depend on the oracle to make an orderly exit. notice the very un-corporate-like nature of this proposal. it involves taking away the governance's ability to act quickly, and intentionally spreading out oracle responsibility across a large number of participants. this is valuable for two reasons. first, it makes it harder for outsiders to attack the oracle, and for new coin holders to quickly take over control of the oracle. second, it makes it harder for the oracle participants themselves to collude to attack the system. it also mitigates oracle extractable value, where a single provider might intentionally delay publishing to personally profit from a liquidation (in a multi-provider system, if one provider doesn't immediately publish, others soon will). fairness in kleros the "decentralized court" system kleros is a really valuable and important piece of infrastructure for the ethereum ecosystem: proof of humanity uses it, various "smart contract bug insurance" products use it, and many other projects plug into it as some kind of "adjudication of last resort". recently, there have been some public concerns about whether or not the platform's decision-making is fair. some participants have made cases, trying to claim a payout from decentralized smart contract insurance platforms that they argue they deserve. perhaps the most famous of these cases is mizu's report on case #1170. the case blew up from being a minor language intepretation dispute into a broader scandal because of the accusation that insiders to kleros itself were making a coordinated effort to throw a large number of tokens to pushing the decision in the direction they wanted. a participant to the debate writes: the incentives-based decision-making process of the court ... is by all appearances being corrupted by a single dev with a very large (25%) stake in the courts. of course, this is but one side of one issue in a broader debate, and it's up to the kleros community to figure out who is right or wrong and how to respond. but zooming out from the question of this individual case, what is important here is the the extent to which the entire value proposition of something like kleros depends on it being able to convince the public that it is strongly protected against this kind of centralized manipulation. for something like kleros to be trusted, it seems necessary that there should not be a single individual with a 25% stake in a high-level court. whether through a more widely distributed token supply, or through more use of non-token-driven governance, a more credibly decentralized form of governance could help kleros avoid such concerns entirely. optimism retro funding optimism's retroactive founding round 1 results were chosen by a quadratic vote among 24 "badge holders". round 2 will likely use a larger number of badge holders, and the eventual goal is to move to a system where a much larger body of citizens control retro funding allocation, likely through some multilayered mechanism involving sortition, subcommittees and/or delegation. there have been some internal debates about whether to have more vs fewer citizens: should "citizen" really mean something closer to "senator", an expert contributor who deeply understands the optimism ecosystem, should it be a position given out to just about anyone who has significantly participated in the optimism ecosystem, or somewhere in between? my personal stance on this issue has always been in the direction of more citizens, solving governance inefficiency issues with second-layer delegation instead of adding enshrined centralization into the governance protocol. one key reason for my position is the potential for insider trading and self-dealing issues. the optimism retroactive funding mechanism has always been intended to be coupled with a prospective speculation ecosystem: public-goods projects that need funding now could sell "project tokens", and anyone who buys project tokens becomes eligible for a large retroactively-funded compensation later. but this mechanism working well depends crucially on the retroactive funding part working correctly, and is very vulnerable to the retroactive funding mechanism becoming corrupted. some example attacks: if some group of people has decided how they will vote on some project, they can buy up (or if overpriced, short) its project token ahead of releasing the decision. if some group of people knows that they will later adjudicate on some specific project, they can buy up the project token early and then intentionally vote in its favor even if the project does not actually deserve funding. funding deciders can accept bribes from projects. there are typically three ways of dealing with these types of corruption and insider trading issues: retroactively punish malicious deciders. proactively filter for higher-quality deciders. add more deciders. the corporate world typically focuses on the first two, using financial surveillance and judicious penalties for the first and in-person interviews and background checks for the second. the decentralized world has less access to such tools: project tokens are likely to be tradeable anonymously, daos have at best limited recourse to external judicial systems, and the remote and online nature of the projects and the desire for global inclusivity makes it harder to do background checks and informal in-person "smell tests" for character. hence, the decentralized world needs to put more weight on the third technique: distribute decision-making power among more deciders, so that each individual decider has less power, and so collusions are more likely to be whistleblown on and revealed. should daos learn more from corporate governance or political science? curtis yarvin, an american philosopher whose primary "big idea" is that corporations are much more effective and optimized than governments and so we should improve governments by making them look more like corporations (eg. by moving away from democracy and closer to monarchy), recently wrote an article expressing his thoughts on how dao governance should be designed. not surprisingly, his answer involves borrowing ideas from governance of traditional corporations. from his introduction: instead the basic design of the anglo-american limited-liability joint-stock company has remained roughly unchanged since the start of the industrial revolution—which, a contrarian historian might argue, might actually have been a corporate revolution. if the joint-stock design is not perfectly optimal, we can expect it to be nearly optimal. while there is a categorical difference between these two types of organizations—we could call them first-order (sovereign) and second-order (contractual) organizations—it seems that society in the current year has very effective second-order organizations, but not very effective first-order organizations. therefore, we probably know more about second-order organizations. so, when designing a dao, we should start from corporate governance, not political science. yarvin's post is very correct in identifying the key difference between "first-order" (sovereign) and "second-order" (contractual) organizations in fact, that exact distinction is precisely the topic of the section in my own post above on credible fairness. however, yarvin's post makes a big, and surprising, mistake immediately after, by immediately pivoting to saying that corporate governance is the better starting point for how daos should operate. the mistake is surprising because the logic of the situation seems to almost directly imply the exact opposite conclusion. because daos do not have a sovereign above them, and are often explicitly in the business of providing services (like currency and arbitration) that are typically reserved for sovereigns, it is precisely the design of sovereigns (political science), and not the design of corporate governance, that daos have more to learn from. to yarvin's credit, the second part of his post does advocate an "hourglass" model that combines a decentralized alignment and accountability layer and a centralized management and execution layer, but this is already an admission that dao design needs to learn at least as much from first-order orgs as from second-order orgs. sovereigns are inefficient and corporations are efficient for the same reason why number theory can prove very many things but abstract group theory can prove much fewer things: corporations fail less and accomplish more because they can make more assumptions and have more powerful tools to work with. corporations can count on their local sovereign to stand up to defend them if the need arises, as well as to provide an external legal system they can lean on to stabilize their incentive structure. in a sovereign, on the other hand, the biggest challenge is often what to do when the incentive structure is under attack and/or at risk of collapsing entirely, with no external leviathan standing ready to support it. perhaps the greatest problem in the design of successful governance systems for sovereigns is what samo burja calls "the succession problem": how to ensure continuity as the system transitions from being run by one group of humans to another group as the first group retires. corporations, burja writes, often just don't solve the problem at all: silicon valley enthuses over "disruption" because we have become so used to the succession problem remaining unsolved within discrete institutions such as companies. daos will need to solve the succession problem eventually (in fact, given the sheer frequency of the "get rich and retire" pattern among crypto early adopters, some daos have to deal with succession issues already). monarchies and corporate-like forms often have a hard time solving the succession problem, because the institutional structure gets deeply tied up with the habits of one specific person, and it either proves difficult to hand off, or there is a very-high-stakes struggle over whom to hand it off to. more decentralized political forms like democracy have at least a theory of how smooth transitions can happen. hence, i would argue that for this reason too, daos have more to learn from the more liberal and democratic schools of political science than they do from the governance of corporations. of course, daos will in some cases have to accomplish specific complicated tasks, and some use of corporate-like forms for accomplishing those tasks may well be a good idea. additionally, daos need to handle unexpected uncertainty. a system that was intended to function in a stable and unchanging way around one set of assumptions, when faced with an extreme and unexpected change to those circumstances, does need some kind of brave leader to coordinate a response. a prototypical example of the latter is stablecoins handling a us dollar collapse: what happens when a stablecoin dao that evolved around the assumption that it's just trying to track the us dollar suddenly faces a world where the us dollar is no longer a viable thing to be tracking, and a rapid switch to some kind of cpi is needed? stylized diagram of the internal experience of the rai ecosystem going through an unexpected transition to a cpi-based regime if the usd ceases to be a viable reference asset. here, corporate governance-inspired approaches may seem better, because they offer a ready-made pattern for responding to such a problem: the founder organizes a pivot. but as it turns out, the history of political systems also offers a pattern well-suited to this situation, and one that covers the question of how to go back to a decentralized mode when the crisis is over: the roman republic custom of electing a dictator for a temporary term to respond to a crisis. realistically, we probably only need a small number of daos that look more like constructs from political science than something out of corporate governance. but those are the really important ones. a stablecoin does not need to be efficient; it must first and foremost be stable and decentralized. a decentralized court is similar. a system that directs funding for a particular cause whether optimism retroactive funding, vitadao, ukrainedao or something else is optimizing for a much more complicated purpose than profit maximization, and so an alignment solution other than shareholder profit is needed to make sure it keeps using the funds for the purpose that was intended. by far the greatest number of organizations, even in a crypto world, are going to be "contractual" second-order organizations that ultimately lean on these first-order giants for support, and for these organizations, much simpler and leader-driven forms of governance emphasizing agility are often going to make sense. but this should not distract from the fact that the ecosystem would not survive without some non-corporate decentralized forms keeping the whole thing stable. dark mode toggle 不同類型的 zk-evm 2022 aug 29 see all posts 特別感謝 pse, polygon hermez, zksync, scroll, matter labs and starkware 團隊參與討論和協助校稿 近來各家 zk-evm 紛紛登場。polygon 的 zk-evm 以開源的方式釋出，zksync 公佈了 zksync 2.0 的規劃，相對晚進場的 scroll 也推出了他們的 zk-evm。正在開發 zk-evm 的，還有 pse, nicholas liochon 等人，以及 starkware 正在開發的 alpha compiler (能把 evm 編譯成 starkware 所開發的 cairo)，族繁不及備載。 上述專案有一個共同的目標：利用 zk-snark 進行密碼學證明，驗證以太坊生態（ethereum-like）交易的執行。這樣更好驗證以太坊 l1 鏈上的交易與狀態，也建立（接近）與以太等價且擴展性更好的 zk-rollups。但這幾個專案之間的些微的差異，反映在實用性和速度之間的取捨。本文試著提出分類不同 evm 的方法，並說明其中的利弊得失。 懶人包（以圖表呈現） zkevm 第 1 類：等價以太坊（fully ethereum-equivalent）的 zk-evm 第 1 類 zk-evm 志在完整、毫無妥協的達到以太坊等價。它沒有改動以太坊生態任何部份的設計以讓證明更簡單，也沒有換掉雜湊的運算（hashes）、狀態樹（state trees）、交易樹（transaction trees）、預編譯的合約（precompiles)，而且再怎麼邊陲的共識邏輯（in-consensus logic），都沒有換掉。 優點：完美的相容性 一切的用意在於用現況下的以太鏈—至少，要驗證執行層（也因此不包含信標鏈（beacon chain）的共識邏輯，但是所有交易執行和智能合約、帳戶的概念，都有涵蓋）。 第 1 類的 zk-evm 是以太坊 l1 更有擴展性的最終解。長遠來看，對以太坊的修正，可能會在第 2 類或第 3 類測試之後，引入到正規以太坊。然後，要作架構的重規劃，則會有其複雜之處。 第 1 類的 zk-evm 也對 rollups 很理想，因為 rollups 能夠重複利用許多的基礎設施。舉例來說，以太坊的執行端可以被用來生成和處理 rollup blocks（退一步來說，等到解除質押存款生效後，這個功能可以被重新用在 rollup 的以太質押存款），所以比如區塊鏈瀏覽器、block production 等工具，都很容易重複利用。 缺點：證明者運算時間 （prover time) 以太坊原生不以零知識證明基礎建構，所以有許多以太坊固有元件，若要作零知識驗證，需要消耗龐大的運算時間。第 1 類 zk-evm 為求完全複製以太坊運作，因此沒有避開低效率的證明流程。就現階段來說，從以太坊既有區塊產生零知識證明，要花上好幾個小時。然而，這個障礙可以透過巧妙的工程設計，大幅提升驗證者平行化產出零知識證明的能力，或建構出 zk-snark asic 等方式，緩解其缺點。 實例 pse 正在蓋的 zk-evm 屬於第 1 類。 第 2 類：等價以太坊虛擬機（fully evm-equivalent）的 zk-evm 第 2 類 zk-evm 試作到跟 evm 等價，但又不完全跟以太坊等價。也就是說，他們「內部」跟以太坊一樣，但是從外面看上去會有些差異，尤其是區塊結構（block structure）、狀態樹（state tree）等資料結構。 一切的用意在於要跟既有的應用軟體完全相容，但是針對以太坊作一些微調，讓開發更容易、生成證明更快速。 優點：等價於虛擬機 第 2 類 zk-evm 改動了儲存諸如 ethereum state 的資料結構。幸運的是，這是都是 evm 本身無法直接存取的結構，所以在 ethereum 之上執行的應用程式幾乎都可以在第 2 類 zk-evm rollup 上使用。你無法直接利用以太的執行端，但經過微調之後可以，evm 的除錯工具，和多數的開發設施，也都能照常使用。 也有少數的例外的情況。對於使用以太坊歷史區塊（historical ethereum blocks）的雜湊數驗證（merkle proofs） 來驗證對歷史交易、交易明細、或狀態（claims about historical transactions, receipts, or state）（例如：跨鏈橋有時候會這麼作）就會有不相容的情況發生。如果有 zk-evm 用其他雜湊函數取代 keccak，這些證明會失效。然而，我本來就不建議這樣設計應用程式，因為未來以太坊會引用的改變（例如 verkle trees）會讓這些應用程式連在以太坊上都不能使用。比較好的替代方案，是以太坊本身應該要新增不容易被未來科技淘汰的歷史取用預編譯合約（future-proof history access precompiles）。 缺點：證明者運算時間稍有改善到仍然很慢 第 2 類 zk-evm 的證明者運算速度，比第 1 類更快，主要的原因，是因為不再使用某些以太坊上，對零知識技術毫無意義地不友善的密碼學。尤其，他們可能會改變 ethereum 的 keccak 和基於 rlp 的 merkle patricia tree，還有或許區塊及交易明細的結構。第 2 類 zk-evm 可能會使用不同的雜湊函數，例如 poseidon。也很自然發生的改變是修正狀態樹，以儲存合約碼雜湊值（code hash）和 keccak，免除對於執行extcodehash和extcodecopy所需要的雜湊驗證。 這些改動大幅的改善證明者運算時間，但不會完全解決所有問題。證明現況下的evm的緩慢效率，以及源自於evm的其他不效率、對zk的不友善，都還留著。舉一個簡單例子：記憶體。因為一個mload只能一次讀 32 bytes，包含「沒有對齊的」字節（起始端和終端都不是32的倍數），一個 mload 無法被直接讀取為一個 chunk，它可能需要讀超過兩個連續的 chuck，並作一些運算，合併結果。 實例 scroll 的 zk-evm 以及 polygon hermez 都屬於第 2 類。然而，兩個專案距離到位都還早。尤其因為還沒加入比較複雜的預編譯，因此，以目前來說，兩個專案都更應該被分類在第 3 類。 第 2.5 類：跟以太坊虛擬機結構一樣，但是 gas 訂價例外 想要大幅減少最壞可能的證明者運算時間（worst-case prover times）的方法，就是針對 evm 內很難的零知識證明運算，大幅提升所需的 gas 訂價。這會牽扯到預編譯、keccak 操作碼、以及或許呼叫合約的特殊模式（specific patterns of calling contracts）、存取記憶體（accessing memory）、 儲存空間（storage）或還原（reverting）。 改善 gas 訂價可能會降低開發者工具的相容性，並會讓一些應用程式不能用（break），但是它比起更「深層」的 evm 改動風險較低。開發者應該要注意，不要在一筆交易中花費超過一個區塊能容納的 gas，也永遠不要把呼叫合約時所需要花用的 gas 寫死（這本來就是會給開發者的標準建議）。 另一個解決資源限制問題的替代方式，是直接對每一個運算可以被呼叫的次數設下硬性的限制。這在電路上更容易實作，但是對 evm 的安全性假設就沒有那麼合適。這種作法我認為屬於第 3 類而非第 2.5 類。 第 3 類：接近以太坊虛擬機等效（almost evm-equivalent） 第 3 類 zk-evm 是接近 evm 等效，只有作了退讓，以改善證明者運算時間，並讓開發更容易。 優點：更容易實作，證明者運算時間縮短 第 3 類 zk-evm 可能會拿掉一些特別難改成 zk-evm 的功能。最有可能被拿掉的，就是預編譯。此外，各種第 3 類 zk-evm，在處理合約程式碼（contract code）、記憶體（memory）或堆疊（stack）上，也有些微差異。 缺點：不相容性 第 3 類 zk-evm 的目標是跟多數的應用程式相容，並且只需要重寫極少部分。即便如此，還是有一些需要重寫的應用程式，因為他們要麼使用了第 3 類 zk-evm 拿掉的預編譯功能，或是在特殊情況（edge cases）之下，某些之間虛擬機處理方法不同的相依性問題（dependencies）。 實例 scroll 和 polygon，雖然它們預期會在未來改善相容性，但目前都屬於第 3 類。polygon 有一個特別設計，是要驗證自己的語言，zkasm，而他們編譯 zk-evm 程式用的是 zkasm。雖然如此，我還是認為這是第 3 類 zk-evm。它仍能驗證evm程式碼，只是換了內部邏輯。 現況下，沒有哪個 zk-evm 團隊是想要成為第 3 類 zk-evm。第 3 類只是一個過渡期，正在等待預編譯功能完成，再換到第 2.5 類。但是在未來，第 1 或 2 類可能會加入 zk-snark 友善的預編譯，自發的變成第 3 類，讓證明者運算時間和 gas 都降低。 第 4 類（等價高階語言（high-level-language equivalent）） 第 4 類把（例如：solidity、vyper，或其他這兩種語言都會編譯成為的中間語言）然後把高階語言編譯成其他有特別設計過、對 zk-snark 友善的語言。 優點：證明者所需運算時間非常短 只要從高階語言就有利用零知識證明，而非等到執行階段的各個步驟，才開始用在 evm 上，就可以省掉很多麻煩。 我雖然只用了一句話描述這個優點（但下面列出那麼多相容性的缺點），但我並不是要說第 4 類沒什麼優點。這是個非常大的優點！直接從高階語言編譯真的可以大幅降低成本，也因為參與證明的門檻變低，所以更去中心化。 缺點：相容性低 通常一個「一般」的，由 vyper 或 solidity 寫成的應用程式可以編譯然後就能用了，但是有很多應用程式沒辦法那麼「一般」，而且理由很重要： 因為在 create2 合約地址要看 bytecode 決定，所以 「第 4 類系統中的合約地址」與「evm 的合約地址」不一定相同。這會造成很多應用程式不能使用，例如倚賴尚未部署的「反事實合約（counterfactual contracts）」的系統、erc-4337 錢包、eip-2470 singletons、等應用程式。 手刻的 evm bytecode 更難被利用。許多程式某些部分會用手刻 evm bytecode 來提高效率。第 4 類的系統就不支援。雖然有些方法可以實作有限的 evm bytecode support 來滿足這些情況，不一定要直上第 3 類 zk-evm。 很多除錯的基礎設施無法被直接利用，因為這些基礎設施只能跑在 evm bytecode 上。話雖如此，有很多來自傳統高階或中階語言（例如 llvm）的除錯工具可以用來緩解這個缺點。 以上都是工程師需要留意的問題。 實例 zksync 是第 4 類系統，雖然隨著時間他們可能會改善對 evm bytecode 的相容性。nethermind 的 warp 專案正在蓋一個能把 solidity 編譯成 starkware 的 cairo 語言的編譯器，這會因此把 starknet 在實然上變成第 4 類系統。 zk-evm 類型的未來展望 以上不同類型之間沒有誰更「好」或更「壞」，而是取捨的不同：越靠近 第 1 類的類型跟既有的基礎設施相容性更高，但是更慢。越靠近第 4 類的類型相容性比較差，但是速度更快。通常來說，對整個生態比較好的作法，就是每種不同類型，都有人研究。 靠近第 4 類的 zk-evm 類型，也可以隨著時間，再慢慢改成靠近第 1 類的類型。反之亦然。舉例來說： zk-evm 可以先跳過很難用零知識證明的功能，當第 3 類就好。之後，再慢慢把功能加上去，逐漸轉成第 2 類。 zk-evm 也可以先當第 2 類，然後之後提供在以太坊完全等效的環境中，或是證明速度更快、改動過的狀態樹，變成第 2 和第 1 類的綜合體。scroll 正在考慮用這個策略發展。 原本是第 4 類的系統，也可以慢慢加上處理 evm opcode 的能力（雖然開發者仍然被鼓勵直接從高階語言編譯，以節省手續費和改善證明者運算所需時間）。 假設以太坊本身變得更零知識友善，原本是第 2 類或第 3 類開始的zk-evm，就會變成第 1 類。 如果在第 1 類或第 2 類 zk-evm 預編譯元件（precompiles），這樣就能使其變成第 3 類 zk-evm。這些預編譯元件的驗證效率很高，是由一種 zk-snark 友善的語言撰寫而成。這樣開發者就能在以太坊的相容性和速度之間能作選擇。這種 zk-evm 屬於第 3 類，因為它犧牲了完美以太坊等價性，但是從實用性的角度來看，它比起第 1 類或第 2 類有更多好處。主要的缺失在於某些開發者工具不支援 zk-evm 客製化的預編譯，然而，這可以被改善：透過開發者手動指定設定檔，開發者工具可以支援將預編譯元件轉換回同等功能的 evm 程式碼，這樣就能適用於所有環境。 我個人的希望，是隨著時間，zk-evm 技術的進步，以及以太坊本身對 zk-snark 的設計更友善之後，所有的zk-evm能漸漸發展為第 1 類。在這樣的未來，我們會有好幾個能夠同時作為 zk-rollup 以及驗證以太鏈本身的 zk-evm。理論上，沒有必要將以太坊標準化為只能供一種 zk-evm 使用的 l1。要有不同的客戶端使用不同的證明方式，我們才能收穫冗餘實作（code redundancy）的好處。 然而，要花一些時間，才會抵達這樣的未來。在這條路上，將會看到很多擴展以太坊、以太坊 zk-rollups 技術的日新月異。 dark mode toggle endgame 2021 dec 06 see all posts special thanks to a whole bunch of people from optimism and flashbots for discussion and thought that went into this piece, and karl floersch, phil daian, hasu and alex obadia for feedback and review. consider the average "big block chain" very high block frequency, very high block size, many thousands of transactions per second, but also highly centralized: because the blocks are so big, only a few dozen or few hundred nodes can afford to run a fully participating node that can create blocks or verify the existing chain. what would it take to make such a chain acceptably trustless and censorship resistant, at least by my standards? here is a plausible roadmap: add a second tier of staking, with low resource requirements, to do distributed block validation. the transactions in a block are split into 100 buckets, with a merkle or verkle tree state root after each bucket. each second-tier staker gets randomly assigned to one of the buckets. a block is only accepted when at least 2/3 of the validators assigned to each bucket sign off on it. introduce either fraud proofs or zk-snarks to let users directly (and cheaply) check block validity. zk-snarks can cryptographically prove block validity directly; fraud proofs are a simpler scheme where if a block has an invalid bucket, anyone can broadcast a fraud proof of just that bucket. this provides another layer of security on top of the randomly-assigned validators. introduce data availability sampling to let users check block availability. by using das checks, light clients can verify that a block was published by only downloading a few randomly selected pieces. add secondary transaction channels to prevent censorship. one way to do this is to allow secondary stakers to submit lists of transactions which the next main block must include. what do we get after all of this is done? we get a chain where block production is still centralized, but block validation is trustless and highly decentralized, and specialized anti-censorship magic prevents the block producers from censoring. it's somewhat aesthetically ugly, but it does provide the basic guarantees that we are looking for: even if every single one of the primary stakers (the block producers) is intent on attacking or censoring, the worst that they could do is all go offline entirely, at which point the chain stops accepting transactions until the community pools their resources and sets up one primary-staker node that is honest. now, consider one possible long-term future for rollups... imagine that one particular rollup whether arbitrum, optimism, zksync, starknet or something completely new does a really good job of engineering their node implementation, to the point where it really can do 10,000 transactions per second if given powerful enough hardware. the techniques for doing this are in-principle well-known, and implementations were made by dan larimer and others many years ago: split up execution into one cpu thread running the unparallelizable but cheap business logic and a huge number of other threads running the expensive but highly parallelizable cryptography. imagine also that ethereum implements sharding with data availability sampling, and has the space to store that rollup's on-chain data between its 64 shards. as a result, everyone migrates to this rollup. what would that world look like? once again, we get a world where, block production is centralized, block validation is trustless and highly decentralized, and censorship is still prevented. rollup block producers have to process a huge number of transactions, and so it is a difficult market to enter, but they have no way to push invalid blocks through. block availability is secured by the underlying chain, and block validity is guaranteed by the rollup logic: if it's a zk rollup, it's ensured by snarks, and an optimistic rollup is secure as long as there is one honest actor somewhere running a fraud prover node (they can be subsidized with gitcoin grants). furthermore, because users always have the option of submitting transactions through the on-chain secondary inclusion channel, rollup sequencers also cannot effectively censor. now, consider the other possible long-term future of rollups... no single rollup succeeds at holding anywhere close to the majority of ethereum activity. instead, they all top out at a few hundred transactions per second. we get a multi-rollup future for ethereum the cosmos multi–chain vision, but on top of a base layer providing data availability and shared security. users frequently rely on cross-rollup bridging to jump between different rollups without paying the high fees on the main chain. what would that world look like? it seems like we could have it all: decentralized validation, robust censorship resistance, and even distributed block production, because the rollups are all invididually small and so easy to start producing blocks in. but the decentralization of block production may not last, because of the possibility of cross-domain mev. there are a number of benefits to being able to construct the next block on many domains at the same time: you can create blocks that use arbitrage opportunities that rely on making transactions in two rollups, or one rollup and the main chain, or even more complex combinations. a cross-domain mev opportunity discovered by western gate hence, in a multi-domain world, there are strong pressures toward the same people controlling block production on all domains. it may not happen, but there's a good chance that it will, and we have to be prepared for that possibility. what can we do about it? so far, the best that we know how to do is to use two techniques in combination: rollups implement some mechanism for auctioning off block production at each slot, or the ethereum base layer implements proposer/builder separation (pbs) (or both). this ensures that at least any centralization tendencies in block production don't lead to a completely elite-captured and concentrated staking pool market dominating block validation. rollups implement censorship-resistant bypass channels, and the ethereum base layer implements pbs anti-censorship techniques. this ensures that if the winners of the potentially highly centralized "pure" block production market try to censor transactions, there are ways to bypass the censorship. so what's the result? block production is centralized, block validation is trustless and highly decentralized, and censorship is still prevented. three paths toward the same destination. so what does this mean? while there are many paths toward building a scalable and secure long-term blockchain ecosystem, it's looking like they are all building toward very similar futures. there's a high chance that block production will end up centralized: either the network effects within rollups or the network effects of cross-domain mev push us in that direction in their own different ways. but what we can do is use protocol-level techniques such as committee validation, data availability sampling and bypass channels to "regulate" this market, ensuring that the winners cannot abuse their power. what does this mean for block producers? block production is likely to become a specialized market, and the domain expertise is likely to carry over across different domains. 90% of what makes a good optimism block producer also makes a good arbitrum block producer, and a good polygon block producer, and even a good ethereum base layer block producer. if there are many domains, cross-domain arbitrage may also become an important source of revenue. what does this mean for ethereum? first of all, ethereum is very well-positioned to adjust to this future world, despite the inherent uncertainty. the profound benefit of the ethereum rollup-centric roadmap is that it means that ethereum is open to all of the futures, and does not have to commit to an opinion about which one will necessarily win. will users very strongly want to be on a single rollup? ethereum, following its existing course, can be the base layer of that, automatically providing the anti-fraud and anti-censorship "armor" that high-capacity domains need to be secure. is making a high-capacity domain too technically complicated, or do users just have a great need for variety? ethereum can be the base layer of that too and a very good one, as the common root of trust makes it far easier to move assets between rollups safely and cheaply. but also, ethereum researchers should think hard about what levels of decentralization in block production are actually achievable. it may not be worth it to add complicated plumbing to make highly decentralized block production easy if cross-domain mev (or even cross-shard mev from one rollup taking up multiple shards) make it unsustainable regardless. what does this mean for big block chains? there is a path for them to turn into something trustless and censorship resistant, and we'll soon find out if their core developers and communities actually value censorship resistance and decentralization enough for them to do it! it will likely take years for all of this to play out. sharding and data availability sampling are complex technologies to implement. it will take years of refinement and audits for people to be fully comfortable storing their assets in a zk-rollup running a full evm. and cross-domain mev research too is still in its infancy. but it does look increasingly clear how a realistic but bright future for scalable blockchains is likely to emerge. jutta’s update on bug bounty program and security audit | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search jutta’s update on bug bounty program and security audit posted by jutta steiner on march 20, 2015 research & development hi, jutta writing again – i initially introduced myself when we started the bounty program earlier this year and i’m happy to provide you with an update on what’s happening on the security side prior and throughout launch. we have had some high quality submissions by bounty hunters – kudos for the creative exploits sent along that the bugs made possible. the number of submissions recently rose. hence we decided and are announcing that we will continue the bounty program at least throughout the frontier phase of ethereum’s launch plan – see gav's and vinay's respective blog posts. please go to our bounty website for more information on the bounty program and make sure check out our lead hunter’s repository here for helpful testing scripts before starting the hunt. not only do we rely on individual bug hunters and the community: together with ethdev’s  gustav simonsson, i had started the process of selecting professional security experts, academics and blockchain experts for our external security audits late last year. gustav is now working with auditors and the ethereum go dev team to track all security issues tagged here and work out fixes for them. we want to keep good track of all issues and only close them once fully resolved and solutions sufficiently tested. every bug we find is taken care of and will be fixed before frontier release. feel free to follow us on github if you want to keep an eye on the progress. the first round of work from security auditors is finishing in a couple of weeks, and bug fixing is already well underway. working through all issues will take the time it takes. it’s a security-driven not schedule-driven process, after all. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle some personal user experiences 2023 feb 28 see all posts in 2013, i went to a sushi restaurant beside the internet archive in san francisco, because i had heard that it accepted bitcoin for payments and i wanted to try it out. when it came time to pay the bill, i asked to pay in btc. i scanned the qr code, and clicked "send". to my surprise, the transaction did not go through; it appeared to have been sent, but the restaurant was not receiving it. i tried again, still no luck. i soon figured out that the problem was that my mobile internet was not working well at the time. i had to walk over 50 meters toward the internet archive nearby to access its wifi, which finally allowed me to send the transaction. lesson learned: internet is not 100% reliable, and customer internet is less reliable than merchant internet. we need in-person payment systems to have some functionality (nfc, customer shows a qr code, whatever) to allow customers to transfer their transaction data directly to the merchant if that's the best way to get it broadcasted. in 2021, i attempted to pay for tea for myself and my friends at a coffee shop in argentina. in their defense, they did not intentionally accept cryptocurrency: the owner simply recognized me, and showed me that he had an account at a cryptocurrency exchange, so i suggested to pay in eth (using cryptocurrency exchange accounts as wallets is a standard way to do in-person payments in latin america). unfortunately, my first transaction of 0.003 eth did not get accepted, probably because it was under the exchange's 0.01 eth deposit minimum. i sent another 0.007 eth. soon, both got confirmed. (i did not mind the 3x overpayment and treated it as a tip). in 2022, i attempted to pay for tea at a different location. the first transaction failed, because the default transaction from my mobile wallet sent with only 21000 gas, and the receiving account was a contract that required extra gas to process the transfer. attempts to send a second transaction failed, because a ui glitch in my phone wallet made it not possible to scroll down and edit the field that contained the gas limit. lesson learned: simple-and-robust uis are better than fancy-and-sleek ones. but also, most users don't even know what gas limits are, so we really just need to have better defaults. many times, there has been a surprisingly long time delay between my transaction getting accepted on-chain, and the service acknowledging the transaction, even as "unconfirmed". some of those times, i definitely got worried that there was some glitch with the payment system on their side. many times, there has been a surprisingly long and unpredictable time delay between sending a transaction, and that transaction getting accepted in a block. sometimes, a transaction would get accepted in a few seconds, but other times, it would take minutes or even hours. recently, eip-1559 significantly improved this, ensuring that most transactions get accepted into the next block, and even more recently the merge improved things further by stabilizing block times. diagram from this report by yinhong (william) zhao and kartik nayak. however, outliers still remain. if you send a transaction at the same time as when many others are sending transactions and the base fee is spiking up, you risk the base fee going too high and your transaction not getting accepted. even worse, wallet uis suck at showing this. there are no big red flashing alerts, and very little clear indication of what you're supposed to do to solve this problem. even to an expert, who knows that in this case you're supposed to "speed up" the transaction by publishing a new transaction with identical data but a higher max-basefee, it's often not clear where the button to do that actually is. lesson learned: ux around transaction inclusion needs to be improved, though there are fairly simple fixes. credit to the brave wallet team for taking my suggestions on this topic seriously, and first increasing the max-basefee tolerance from 12.5% to 33%, and more recently exploring ways to make stuck transactions more obvious in the ui. in 2019, i was testing out one of the earliest wallets that was attempting to provide social recovery. unlike my preferred approach, which is smart-contract-based, their approach was to use shamir's secret sharing to split up the private key to the account into five pieces, in such a way that any three of those pieces could be used to recover the private key. users were expected to choose five friends ("guardians" in modern lingo), convince them to download a separate mobile application, and provide a confirmation code that would be used to create an encrypted connection from the user's wallet to the friend's application through firebase and send them their share of the key. this approach quickly ran into problems for me. a few months later, something happened to my wallet and i needed to actually use the recovery procedure to recover it. i asked my friends to perform the recovery procedure with me through their apps but it did not go as planned. two of them lost their key shards, because they switched phones and forgot to move the recovery application over. for a third, the firebase connection mechanism did not work for a long time. eventually, we figured out how to fix the issue, and recover the key. a few months after that, however, the wallet broke again. this time, a regular software update somehow accidentally reset the app's storage and deleted its key. but i had not added enough recovery partners, because the firebase connection mechanism was too broken and was not letting me successfully do that. i ended up losing a small amount of btc and eth. lesson learned: secret-sharing-based off-chain social recovery is just really fragile and a bad idea unless there are no other options. your recovery guardians should not have to download a separate application, because if you have an application only for an exceptional situation like recovery, it's too easy to forget about it and lose it. additionally, requiring separate centralized communication channels comes with all kinds of problems. instead, the way to add guardians should be to provide their eth address, and recovery should be done by smart contract, using erc-4337 account abstraction wallets. this way, the guardians would only need to not lose their ethereum wallets, which is something that they already care much more about not losing for other reasons. in 2021, i was attempting to save on fees when using tornado cash, by using the "self-relay" option. tornado cash uses a "relay" mechanism where a third party pushes the transaction on-chain, because when you are withdrawing you generally do not yet have coins in your withdrawal address, and you don't want to pay for the transaction with your deposit address because that creates a public link between the two addresses, which is the whole problem that tornado cash is trying to prevent. the problem is that the relay mechanism is often expensive, with relays charging a percentage fee that could go far above the actual gas fee of the transaction. to save costs, one time i used the relay for a first small withdrawal that would charge lower fees, and then used the "self-relay" feature in tornado cash to send a second larger withdrawal myself without using relays. the problem is, i screwed up and accidentally did this while logged in to my deposit address, so the deposit address paid the fee instead of the withdrawal address. oops, i created a public link between the two. lesson learned: wallet developers should start thinking much more explicitly about privacy. also, we need better forms of account abstraction to remove the need for centralized or even federated relays, and commoditize the relaying role. miscellaneous stuff many apps still do not work with the brave wallet or the status browser; this is likely because they didn't do their homework properly and rely on metamask-specific apis. even gnosis safe did not work with these wallets for a long time, leading me to have to write my own mini javascript dapp to make confirmations. fortunately, the latest ui has fixed this issue. the erc20 transfers pages on etherscan (eg. https://etherscan.io/address/0xd8da6bf26964af9d7eed9e03e53415d37aa96045#tokentxns) are very easy to spam with fakes. anyone can create a new erc20 token with logic that can issue a log that claims that i or any other specific person sent someone else tokens. this is sometimes used to trick people into thinking that i support some scam token when i actually have never even heard of it. uniswap used to offer the functionality of being able to swap tokens and have the output sent to a different address. this was really convenient for when i have to pay someone in usdc but i don't have any already on me. now, the interface doesn't offer that function, and so i have to convert and then send in a separate transaction, which is less convenient and wastes more gas. i have since learned that cowswap and paraswap offer the functionality, though paraswap... currently does not seem to work with the brave wallet. sign in with ethereum is great, but it's still difficult to use if you are trying to sign in on multiple devices, and your ethereum wallet is only available on one device. conclusions good user experience is not about the average case, it is about the worst case. a ui that is clean and sleek, but does some weird and unexplainable thing 0.723% of the time that causes big problems, is worse than a ui that exposes more gritty details to the user but at least makes it easier to understand what's going on and fix any problem that does arise. along with the all-important issue of high transaction fees due to scaling not yet being fully solved, user experience is a key reason why many ethereum users, especially in the global south, often opt for centralized solutions instead of on-chain decentralized alternatives that keep power in the hands of the user and their friends and family or local community. user experience has made great strides over the years in particular, going from an average transaction taking minutes to get included before eip-1559 to an average transaction taking seconds to get included after eip-1559 and the merge, has been a night-and-day change to how pleasant it is to use ethereum. but more still needs to be done. dark mode toggle 我在加密世界的一些个人体验 2022 oct 28 see all posts 译者：wonder @ greenpill cn 原文：some personal user experiences 我在加密世界的一些个人体验 2013 年，我去了旧金山互联网档案馆旁边的一家寿司店，因为我听说它接受比特币支付，我想去试试。在结账的时候，我要求用 btc 进行支付，我扫描了二维码，然后点击了「发送」。令我惊讶的是，交易并没有成功；看起来交易已经发送出去，但餐厅并没有收到这笔交易。我又试了一次，还是没有成功。我很快就发现了问题所在，原因在于当时我的网络信号不好。我不得不向附近的互联网档案馆走 50 多米，接入了它的 wifi，才终于将这笔交易发送出去。 经验教训：互联网不是 100% 可靠的，用户互联网比商家互联网更不可靠。线下支付系统需要具备一些能让客户直接将交易数据传输给商家的功能（例如 nfc、客户出示二维码等等），假如这是传输的最佳方式的话。 2021 年，在阿根廷的一家咖啡店，我尝试使用加密货币为自己和朋友支付茶水费。店主表示，他们本不接受加密货币支付，只是因为他认出了我。他向我展示了他在加密货币交易所拥有的一个账户，所以我提议用 eth 进行支付（在拉丁美洲使用加密货币交易所账户作为钱包是一种标准的面对面支付方式）。不凑巧的是，第一笔 0.003 eth 的交易没有被接收成功，可能是因为它低于交易所 0.01 eth 的最低存款额要求。于是我又发送了 0.007 eth，很快，两笔交易都得到了确认。(我并不介意多付 3 倍的钱，就当作是小费吧）。 2022 年，在另一家店铺，我再次尝试用加密货币支付茶水费。第一笔交易失败了，因为我的手机钱包的默认交易只发送了 21,000 gas，而接收账户是一个智能合约，需要额外 gas 才能处理转账。试图发送的第二笔交易也失败了，因为我的手机钱包的用户界面出现了点小故障，无法向下滑动并编辑「gas 限制」这一字段。 经验教训：简单实用的用户界面要优于花哨华丽的用户界面。但是，大多数用户甚至不知道「gas 限制」是什么，所以我们的确需要优化默认值。 有很多次，我的交易被链上接收和系统确认交易之间的延迟出乎意料地久，甚至出现了「未确认」的状态。有时，我的确会担心链上的支付系统是否出现了一些故障。 还有很多次，发送交易和区块接收交易之间存在延迟，并且这种延迟是不可预测的。有时，一笔交易会在几秒钟内被接收，但更多时候一笔交易需要几分钟甚至几小时才被接收。最近，eip-1559 大大优化了这一点，确保大多数交易被接收到下一个区块中，最近的合并通过稳定区块时间进一步改善了这种情况。 yinhong (william) zhao 和 kartik nayak 制作的报告 但是，异常值仍然存在。如果同时有许多人正在发送交易，那么基础 gas 费就会飙升，你就会面临基础 gas 费过高和交易不被接收的风险。更糟糕的是，钱包的用户界面对这种情况的提示也不够到位，既没有显眼的红色闪烁警报，也没有清晰的提醒告知用户应该做什么来解决这个问题。即使你是专家，知道在这种情况下，可以通过提高 gas 费并发布一个数据相同的新交易来「加速」交易，但往往找不到可操作的按钮在哪里。 经验教训：交易相关的用户体验仍需改进，改善方法其实相当简单。感谢 brave 钱包团队认真对待了我在这个问题上的建议，首先将最大基本费用的限制从 12.5% 提高到了 33%；近期，他们还在探索如何在用户界面清晰地展示「交易被卡住了」这一状态的方法。 2019 年，我测试了一款早期的钱包，它提供社交恢复方案。与我偏爱的智能合约方案不同的是，社交恢复采取了沙米尔的秘密分享理念，即将账户的私钥分成五部分，集齐其中任何三个部分都可以恢复私钥。用户需要选择五个朋友（或者是当代术语「守护人（guardians）」），说服他们下载一个单独的移动应用程序，并提供一个确认码，通过 firebase 创建一个从用户钱包到应用程序的加密连接，并将他们拥有的密钥发送给用户。 于我而言，使用这种方法很快就遇到了麻烦。几个月后，我的钱包出了问题，我需要使用恢复程序来恢复钱包。我请朋友们通过他们的应用程序和我一起执行恢复程序，但事情并没有按计划进行。其中两位朋友丢失了他们的密钥，因为他们换了手机，忘记迁移恢复程序。第三位朋友表示，firebase 的连接机制在很长一段时间内不起作用。虽然最终我们还是想出了解决问题的办法，并恢复了密钥。但是，几个月之后，钱包又坏了。这一次的问题在于，一次常规的软件更新莫名其妙重置了应用程序的存储，并删除了密钥。由于 firebase 的连接机制过于垃圾，我无法添加足够的恢复伙伴，最终损失了少量的 btc 和 eth。 经验教训：基于秘密分享的链下社交恢复实在是太脆弱了，在有其他选择的情况下，这绝不是一个好的方法。帮助恢复密钥的守护人不应该下载一个单独的应用程序，因为如果一个应用程序只有「恢复（密钥）」这一种特殊用途，人们很容易忘记和丢失。此外，独立的中心化通信渠道也会带来各种问题。相反，添加守护人的方式应该是提供他们的 eth 地址，恢复过程应该由智能合约完成，例如使用 erc-4337 账户抽象钱包。这样一来，守护人只需保证不丢失以太坊钱包即可；出于其他原因，他们也会更加小心翼翼，不让钱包丢失。 2021 年，我想通过使用 tornado cash 的「自我中继」选项来节省费用。tornado cash 采用的「中继器」机制是由第三方在链上来推送交易。因为当你取款时，你的取款地址中一般没有代币，并且你不想用你的存款地址来支付交易，因为这样会在两个地址之间形成公开的联系，这正是 tornado cash 试图避免的问题。但中继器机制往往很昂贵，中继器收取的百分比费用可能远远超过交易的实际 gas 费。 为了节约成本，有一次，我使用中继器先发起了一笔小额提款，这样收取的费用较低，然后在不使用中继器的情况下，我通过 tornado cash 的「自我中继」功能，发送了第二笔大额提款。然后，我搞砸了，我在登录存款地址时不小心进行了这项操作，所以我使用了存款地址而不是提款地址，支付了费用。糟糕！我在两者之间建立了一个公开的联系。 经验教训：钱包开发者应该开始更明确地考虑隐私问题。此外，我们需要更好的账户抽象形式，以消除对中心化、甚至是联合中继的需求，并使中继角色商品化。 其它事项 许多应用程序不兼容 brave 钱包或 status 浏览器；这可能是因为他们没有好好地做准备工作，并依赖于 metamask 独有的 api。甚至在很长一段时间里，gnosis safe 也不兼容这些钱包，导致我不得不自己编写迷你 javascript dapp 来进行确认。幸运的是，最新的 ui 已经修复了这个问题。 在 etherscan 上的 erc20 转账页面（例如：https://etherscan.io/address/0xd8da6bf26964af9d7eed9e03e53415d37aa96045#tokentxns）非常容易被造假，导致欺诈行为。任何人都可以创建一个新的 erc20 代币，可以发布一个日志，声称我或某个知名人士向其他人发送了代币。有时，人们会被这种把戏欺骗，认为我为一些骗人的代币站台，而实际上我从未听说过它。 uniswap 曾经提供了非常便利的功能，能够交换代币并将输出发送到不同的地址。当我身上没有任何 usdc 时，但我需要用 usdc 支付给某人，这个功能十分便利。现在，uniswap 不提供这个功能了，所以我不得不先完成兑换然后再发送一笔单独的交易，不仅不方便，而且浪费了更多的 gas 费。我后来了解到 cowswap 和 paraswap 提供了这个功能，不过 paraswap......目前似乎不支持 brave 钱包。 使用以太坊登录固然不错，但如果你想在多个设备登录，而你的以太坊钱包只能在一个设备上使用的话，这算不上是好用的方式。 结论 良好的用户体验不是在平均情况下的表现，而是如何处理最坏情况。如果用户界面简约整洁，但在 0.723% 的时间里会出现一些奇怪且无法解释的状况，从而导致一些大麻烦，其实比事无巨细的用户界面更糟糕，因为后者至少可以让用户更容易理解发生了什么，并解决出现的问题。 除了扩容导致的高额交易费用这一重要问题尚未完全解决，用户体验也是许多以太坊用户，尤其是南方世界的用户，经常选择中心化解决方案的关键原因；这也阻碍了用户选择将权力掌握在自己、亲友或当地社区手中这一链上去中心化方案。多年来，用户体验已经取得了长足的进步，尤其是 eip-1559 的采用。在此之前，交易被接收的平均时间是几分钟，而合并之后的平均交易时间只需要几秒钟，这极大提升了使用以太坊的愉悦程度，但以太坊要做的远不止于此。 dark mode toggle moving beyond coin voting governance 2021 aug 16 see all posts special thanks to karl floersch, dan robinson and tina zhen for feedback and review. see also notes on blockchain governance, governance, part 2: plutocracy is still bad, on collusion and coordination, good and bad for earlier thinking on similar topics. one of the important trends in the blockchain space over the past year is the transition from focusing on decentralized finance (defi) to also thinking about decentralized governance (degov). while the 2020 is often widely, and with much justification, hailed as a year of defi, over the year since then the growing complexity and capability of defi projects that make up this trend has led to growing interest in decentralized governance to handle that complexity. there are examples inside of ethereum: yfi, compound, synthetix, uni, gitcoin and others have all launched, or even started with, some kind of dao. but it's also true outside of ethereum, with arguments over infrastructure funding proposals in bitcoin cash, infrastructure funding votes in zcash, and much more. the rising popularity of formalized decentralized governance of some form is undeniable, and there are important reasons why people are interested in it. but it is also important to keep in mind the risks of such schemes, as the recent hostile takeover of steem and subsequent mass exodus to hive makes clear. i would further argue that these trends are unavoidable. decentralized governance in some contexts is both necessary and dangerous, for reasons that i will get into in this post. how can we get the benefits of degov while minimizing the risks? i will argue for one key part of the answer: we need to move beyond coin voting as it exists in its present form. degov is necessary ever since the declaration of independence of cyberspace in 1996, there has been a key unresolved contradiction in what can be called cypherpunk ideology. on the one hand, cypherpunk values are all about using cryptography to minimize coercion, and maximize the efficiency and reach of the main non-coercive coordination mechanism available at the time: private property and markets. on the other hand, the economic logic of private property and markets is optimized for activities that can be "decomposed" into repeated one-to-one interactions, and the infosphere, where art, documentation, science and code are produced and consumed through irreducibly one-to-many interactions, is the exact opposite of that. there are two key problems inherent to such an environment that need to be solved: funding public goods: how do projects that are valuable to a wide and unselective group of people in the community, but which often do not have a business model (eg. layer-1 and layer-2 protocol research, client development, documentation...), get funded? protocol maintenance and upgrades: how are upgrades to the protocol, and regular maintenance and adjustment operations on parts of the protocol that are not long-term stable (eg. lists of safe assets, price oracle sources, multi-party computation keyholders), agreed upon? early blockchain projects largely ignored both of these challenges, pretending that the only public good that mattered was network security, which could be achieved with a single algorithm set in stone forever and paid for with fixed proof of work rewards. this state of affairs in funding was possible at first because of extreme bitcoin price rises from 2010-13, then the one-time ico boom from 2014-17, and again from the simultaneous second crypto bubble of 2014-17, all of which made the ecosystem wealthy enough to temporarily paper over the large market inefficiencies. long-term governance of public resources was similarly ignored: bitcoin took the path of extreme minimization, focusing on providing a fixed-supply currency and ensuring support for layer-2 payment systems like lightning and nothing else, ethereum continued developing mostly harmoniously (with one major exception) because of the strong legitimacy of its pre-existing roadmap (basically: "proof of stake and sharding"), and sophisticated application-layer projects that required anything more did not yet exist. but now, increasingly, that luck is running out, and challenges of coordinating protocol maintenance and upgrades and funding documentation, research and development while avoiding the risks of centralization are at the forefront. the need for degov for funding public goods it is worth stepping back and seeing the absurdity of the present situation. daily mining issuance rewards from ethereum are about 13500 eth, or about $40m, per day. transaction fees are similarly high; the non-eip-1559-burned portion continues to be around 1,500 eth (~$4.5m) per day. so there are many billions of dollars per year going to fund network security. now, what is the budget of the ethereum foundation? about $30-60 million per year. there are non-ef actors (eg. consensys) contributing to development, but they are not much larger. the situation in bitcoin is similar, with perhaps even less funding going into non-security public goods. here is the situation in a chart: within the ethereum ecosystem, one can make a case that this disparity does not matter too much; tens of millions of dollars per year is "enough" to do the needed r&d and adding more funds does not necessarily improve things, and so the risks to the platform's credible neutrality from instituting in-protocol developer funding exceed the benefits. but in many smaller ecosystems, both ecosystems within ethereum and entirely separate blockchains like bch and zcash, the same debate is brewing, and at those smaller scales the imbalance makes a big difference. enter daos. a project that launches as a "pure" dao from day 1 can achieve a combination of two properties that were previously impossible to combine: (i) sufficiency of developer funding, and (ii) credible neutrality of funding (the much-coveted "fair launch"). instead of developer funding coming from a hardcoded list of receiving addresses, the decisions can be made by the dao itself. of course, it's difficult to make a launch perfectly fair, and unfairness from information asymmetry can often be worse than unfairness from explicit premines (was bitcoin really a fair launch considering how few people had a chance to even hear about it by the time 1/4 of the supply had already been handed out by the end of 2010?). but even still, in-protocol compensation for non-security public goods from day one seems like a potentially significant step forward toward getting sufficient and more credibly neutral developer funding. the need for degov for protocol maintenance and upgrades in addition to public goods funding, the other equally important problem requiring governance is protocol maintenance and upgrades. while i advocate trying to minimize all non-automated parameter adjustment (see the "limited governance" section below) and i am a fan of rai's "un-governance" strategy, there are times where governance is unavoidable. price oracle inputs must come from somewhere, and occasionally that somewhere needs to change. until a protocol "ossifies" into its final form, improvements have to be coordinated somehow. sometimes, a protocol's community might think that they are ready to ossify, but then the world throws a curveball that requires a complete and controversial restructuring. what happens if the us dollar collapses, and rai has to scramble to create and maintain their own decentralized cpi index for their stablecoin to remain stable and relevant? here too, degov is necessary, and so avoiding it outright is not a viable solution. one important distinction is whether or not off-chain governance is possible. i have for a long time been a fan of off-chain governance wherever possible. and indeed, for base-layer blockchains, off-chain governance absolutely is possible. but for application-layer projects, and especially defi projects, we run into the problem that application-layer smart contract systems often directly control external assets, and that control cannot be forked away. if tezos's on-chain governance gets captured by an attacker, the community can hard-fork away without any losses beyond (admittedly high) coordination costs. if makerdao's on-chain governance gets captured by an attacker, the community can absolutely spin up a new makerdao, but they will lose all the eth and other assets that are stuck in the existing makerdao cdps. hence, while off-chain governance is a good solution for base layers and some application-layer projects, many application-layer projects, particularly defi, will inevitably require formalized on-chain governance of some form. degov is dangerous however, all current instantiations of decentralized governance come with great risks. to followers of my writing, this discussion will not be new; the risks are much the same as those that i talked about here, here and here. there are two primary types of issues with coin voting that i worry about: (i) inequalities and incentive misalignments even in the absence of attackers, and (ii) outright attacks through various forms of (often obfuscated) vote buying. to the former, there have already been many proposed mitigations (eg. delegation), and there will be more. but the latter is a much more dangerous elephant in the room to which i see no solution within the current coin voting paradigm. problems with coin voting even in the absence of attackers the problems with coin voting even without explicit attackers are increasingly well-understood (eg. see this recent piece by dappradar and monday capital), and mostly fall into a few buckets: small groups of wealthy participants ("whales") are better at successfully executing decisions than large groups of small-holders. this is because of the tragedy of the commons among small-holders: each small-holder has only an insignificant influence on the outcome, and so they have little incentive to not be lazy and actually vote. even if there are rewards for voting, there is little incentive to research and think carefully about what they are voting for. coin voting governance empowers coin holders and coin holder interests at the expense of other parts of the community: protocol communities are made up of diverse constituencies that have many different values, visions and goals. coin voting, however, only gives power to one constituency (coin holders, and especially wealthy ones), and leads to over-valuing the goal of making the coin price go up even if that involves harmful rent extraction. conflict of interest issues: giving voting power to one constituency (coin holders), and especially over-empowering wealthy actors in that constituency, risks over-exposure to the conflicts-of-interest within that particular elite (eg. investment funds or holders that also hold tokens of other defi platforms that interact with the platform in question) there is one major type of strategy being attempted for solving the first problem (and therefore also mitigating the third problem): delegation. smallholders don't have to personally judge each decision; instead, they can delegate to community members that they trust. this is an honorable and worthy experiment; we shall see how well delegation can mitigate the problem. my voting delegation page in the gitcoin dao the problem of coin holder centrism, on the other hand, is significantly more challenging: coin holder centrism is inherently baked into a system where coin holder votes are the only input. the mis-perception that coin holder centrism is an intended goal, and not a bug, is already causing confusion and harm; one (broadly excellent) article discussing blockchain public goods complains: can crypto protocols be considered public goods if ownership is concentrated in the hands of a few whales? colloquially, these market primitives are sometimes described as "public infrastructure," but if blockchains serve a "public" today, it is primarily one of decentralized finance. fundamentally, these tokenholders share only one common object of concern: price. the complaint is false; blockchains serve a public much richer and broader than defi token holders. but our coin-voting-driven governance systems are completely failing to capture that, and it seems difficult to make a governance system that captures that richness without a more fundamental change to the paradigm. coin voting's deep fundamental vulnerability to attackers: vote buying the problems get much worse once determined attackers trying to subvert the system enter the picture. the fundamental vulnerability of coin voting is simple to understand. a token in a protocol with coin voting is a bundle of two rights that are combined into a single asset: (i) some kind of economic interest in the protocol's revenue and (ii) the right to participate in governance. this combination is deliberate: the goal is to align power and responsibility. but in fact, these two rights are very easy to unbundle from each other. imagine a simple wrapper contract that has these rules: if you deposit 1 xyz into the contract, you get back 1 wxyz. that wxyz can be converted back into an xyz at any time, plus in addition it accrues dividends. where do the dividends come from? well, while the xyz coins are inside the wrapper contract, it's the wrapper contract that has the ability to use them however it wants in governance (making proposals, voting on proposals, etc). the wrapper contract simply auctions off this right every day, and distributes the profits among the original depositors. as an xyz holder, is it in your interest to deposit your coins into the contract? if you are a very large holder, it might not be; you like the dividends, but you are scared of what a misaligned actor might do with the governance power you are selling them. but if you are a smaller holder, then it very much is. if the governance power auctioned by the wrapper contract gets bought up by an attacker, you personally only suffer a small fraction of the cost of the bad governance decisions that your token is contributing to, but you personally gain the full benefit of the dividend from the governance rights auction. this situation is a classic tragedy of the commons. suppose that an attacker makes a decision that corrupts the dao to the attacker's benefit. the harm per participant from the decision succeeding is \(d\), and the chance that a single vote tilts the outcome is \(p\). suppose an attacker makes a bribe of \(b\). the game chart looks like this: decision benefit to you benefit to others accept attacker's bribe \(b d * p\) \(-999 * d * p\) reject bribe, vote your conscience \(0\) \(0\) if \(b > d * p\), you are inclined to accept the bribe, but as long as \(b < 1000 * d * p\), accepting the bribe is collectively harmful. so if \(p < 1\) (usually, \(p\) is far below \(1\)), there is an opportunity for an attacker to bribe users to adopt a net-negative decision, compensating each user far less than the harm they suffer. one natural critique of voter bribing fears is: are voters really going to be so immoral as to accept such obvious bribes? the average dao token holder is an enthusiast, and it would be hard for them to feel good about so selfishly and blatantly selling out the project. but what this misses is that there are much more obfuscated ways to separate out profit sharing rights from governance rights, that don't require anything remotely as explicit as a wrapper contract. the simplest example is borrowing from a defi lending platform (eg. compound). someone who already holds eth can lock up their eth in a cdp ("collateralized debt position") in one of these platforms, and once they do that the cdp contract allows them to borrow an amount of xyz up to eg. half the value of the eth that they put in. they can then do whatever they want with this xyz. to recover their eth, they would eventually need to pay back the xyz that they borrowed, plus interest. note that throughout this process, the borrower has no financial exposure to xyz. that is, if they use their xyz to vote for a governance decision that destroys the value of xyz, they do not lose a penny as a result. the xyz they are holding is xyz that they have to eventually pay back into the cdp regardless, so they do not care if its value goes up or down. and so we have achieved unbundling: the borrower has governance power without economic interest, and the lender has economic interest without governance power. there are also centralized mechanisms for separating profit sharing rights from governance rights. most notably, when users deposit their coins on a (centralized) exchange, the exchange holds full custody of those coins, and the exchange has the ability to use those coins to vote. this is not mere theory; there is evidence of exchanges using their users' coins in several dpos systems. the most notable recent example is the attempted hostile takeover of steem, where exchanges used their customers' coins to vote for some proposals that helped to cement a takeover of the steem network that the bulk of the community strongly opposed. the situation was only resolved through an outright mass exodus, where a large portion of the community moved to a different chain called hive. some dao protocols are using timelock techniques to limit these attacks, requiring users to lock their coins and make them immovable for some period of time in order to vote. these techniques can limit buy-then-vote-then-sell attacks in the short term, but ultimately timelock mechanisms can be bypassed by users holding and voting with their coins through a contract that issues a wrapped version of the token (or, more trivially, a centralized exchange). as far as security mechanisms go, timelocks are more like a paywall on a newspaper website than they are like a lock and key. at present, many blockchains and daos with coin voting have so far managed to avoid these attacks in their most severe forms. there are occasional signs of attempted bribes: but despite all of these important issues, there have been much fewer examples of outright voter bribing, including obfuscated forms such as using financial markets, that simple economic reasoning would suggest. the natural question to ask is: why haven't more outright attacks happened yet? my answer is that the "why not yet" relies on three contingent factors that are true today, but are likely to get less true over time: community spirit from having a tightly-knit community, where everyone feels a sense of camaraderie in a common tribe and mission.. high wealth concentration and coordination of token holders; large holders have higher ability to affect the outcome and have investments in long-term relationships with each other (both the "old boys clubs" of vcs, but also many other equally powerful but lower-profile groups of wealthy token holders), and this makes them much more difficult to bribe. immature financial markets in governance tokens: ready-made tools for making wrapper tokens exist in proof-of-concept forms but are not widely used, bribing contracts exist but are similarly immature, and liquidity in lending markets is low. when a small coordinated group of users holds over 50% of the coins, and both they and the rest are invested in a tightly-knit community, and there are few tokens being lent out at reasonable rates, all of the above bribing attacks may perhaps remain theoretical. but over time, (1) and (3) will inevitably become less true no matter what we do, and (2) must become less true if we want daos to become more fair. when those changes happen, will daos remain safe? and if coin voting cannot be sustainably resistant against attacks, then what can? solution 1: limited governance one possible mitigation to the above issues, and one that is to varying extents being tried already, is to put limits on what coin-driven governance can do. there are a few ways to do this: use on-chain governance only for applications, not base layers: ethereum does this already, as the protocol itself is governed through off-chain governance, while daos and other apps on top of this are sometimes (but not always) governed through on-chain governance limit governance to fixed parameter choices: uniswap does this, as it only allows governance to affect (i) token distribution and (ii) a 0.05% fee in the uniswap exchange. another great example is rai's "un-governance" roadmap, where governance has control over fewer and fewer features over time. add time delays: a governance decision made at time t only takes effect at eg. t + 90 days. this allows users and applications that consider the decision unacceptable to move to another application (possibly a fork). compound has a time delay mechanism in its governance, but in principle the delay can (and eventually should) be much longer. be more fork-friendly: make it easier for users to quickly coordinate on and execute a fork. this makes the payoff of capturing governance smaller. the uniswap case is particularly interesting: it's an intended behavior that the on-chain governance funds teams, which may develop future versions of the uniswap protocol, but it's up to users to opt-in to upgrading to those versions. this is a hybrid of on-chain and off-chain governance that leaves only a limited role for the on-chain side. but limited governance is not an acceptable solution by itself; those areas where governance is needed the most (eg. funds distribution for public goods) are themselves among the most vulnerable to attack. public goods funding is so vulnerable to attack because there is a very direct way for an attacker to profit from bad decisions: they can try to push through a bad decision that sends funds to themselves. hence, we also need techniques to improve governance itself... solution 2: non-coin-driven governance a second approach is to use forms of governance that are not coin-voting-driven. but if coins do not determine what weight an account has in governance, what does? there are two natural alternatives: proof of personhood systems: systems that verify that accounts correspond to unique individual humans, so that governance can assign one vote per human. see here for a review of some techniques being developed, and proofofhumanity and brightid and idenanetwork for three attempts to implement this. proof of participation: systems that attest to the fact that some account corresponds to a person that has participated in some event, passed some educational training, or performed some useful work in the ecosystem. see poap for one attempt to implement thus. there are also hybrid possibilities: one example is quadratic voting, which makes the power of a single voter proportional to the square root of the economic resources that they commit to a decision. preventing people from gaming the system by splitting their resource across many identities requires proof of personhood, and the still-existent financial component allows participants to credibly signal how strongly they care about an issue, as well as how strongly they care about the ecosystem. gitcoin quadratic funding is a form of quadratic voting, and quadratic voting daos are being built. proof of participation is less well-understood. the key challenge is that determining what counts as how much participation itself requires a quite robust governance structure. it's possible that the easiest solution involves bootstrapping the system with a hand-picked choice of 10-100 early contributors, and then decentralizing over time as the selected participants of round n determine participation criteria for round n+1. the possibility of a fork helps provide a path to recovery from, and an incentive against, governance going off the rails. proof of personhood and proof of participation both require some form of anti-collusion (see article explaining the issue here and maci documentation here) to ensure that the non-money resource being used to measure voting power remains non-financial, and does not itself end up inside of smart contracts that sell the governance power to the highest bidder. solution 3: skin in the game the third approach is to break the tragedy of the commons, by changing the rules of the vote itself. coin voting fails because while voters are collectively accountable for their decisions (if everyone votes for a terrible decision, everyone's coins drop to zero), each voter is not individually accountable (if a terrible decision happens, those who supported it suffer no more than those who opposed it). can we make a voting system that changes this dynamic, and makes voters individually, and not just collectively, responsible for their decisions? fork-friendliness is arguably a skin-in-the-game strategy, if forks are done in the way that hive forked from steem. in the case that a ruinous governance decision succeeds and can no longer be opposed inside the protocol, users can take it upon themselves to make a fork. furthermore, in that fork, the coins that voted for the bad decision can be destroyed. this sounds harsh, and perhaps it even feels like a violation of an implicit norm that the "immutability of the ledger" should remain sacrosanct when forking a coin. but the idea seems much more reasonable when seen from a different perspective. we keep the idea of a strong firewall where individual coin balances are expected to be inviolate, but only apply that protection to coins that do not participate in governance. if you participate in governance, even indirectly by putting your coins into a wrapper mechanism, then you may be held liable for the costs of your actions. this creates individual responsibility: if an attack happens, and your coins vote for the attack, then your coins are destroyed. if your coins do not vote for the attack, your coins are safe. the responsibility propagates upward: if you put your coins into a wrapper contract and the wrapper contract votes for an attack, the wrapper contract's balance is wiped and so you lose your coins. if an attacker borrows xyz from a defi lending platform, when the platform forks anyone who lent xyz loses out (note that this makes lending the governance token in general very risky; this is an intended consequence). skin-in-the-game in day-to-day voting but the above only works for guarding against decisions that are truly extreme. what about smaller-scale heists, which unfairly favor attackers manipulating the economics of the governance but not severely enough to be ruinous? and what about, in the absence of any attackers at all, simple laziness, and the fact that coin-voting governance has no selection pressure in favor of higher-quality opinions? the most popular solution to these kinds of issues is futarchy, introduced by robin hanson in the early 2000s. votes become bets: to vote in favor of a proposal, you make a bet that the proposal will lead to a good outcome, and to vote against the proposal, you make a bet that the proposal will lead to a poor outcome. futarchy introduces individual responsibility for obvious reasons: if you make good bets, you get more coins, and if you make bad bets you lose your coins. "pure" futarchy has proven difficult to introduce, because in practice objective functions are very difficult to define (it's not just coin price that people want!), but various hybrid forms of futarchy may well work. examples of hybrid futarchy include: votes as buy orders: see ethresear.ch post. voting in favor of a proposal requires making an enforceable buy order to buy additional tokens at a price somewhat lower than the token's current price. this ensures that if a terrible decision succeeds, those who support it may be forced to buy their opponents out, but it also ensures that in more "normal" decisions coin holders have more slack to decide according to non-price criteria if they so wish. retroactive public goods funding: see post with the optimism team. public goods are funded by some voting mechanism retroactively, after they have already achieved a result. users can buy project tokens to fund their project while signaling confidence in it; buyers of project tokens get a share of the reward if that project is deemed to have achieved a desired goal. escalation games: see augur and kleros. value-alignment on lower-level decisions is incentivized by the possibility to appeal to a higher-effort but higher-accuracy higher-level process; voters whose votes agree with the ultimate decision are rewarded. in the latter two cases, hybrid futarchy depends on some form of non-futarchy governance to measure against the objective function or serve as a dispute layer of last resort. however, this non-futarchy governance has several advantages that it does not if used directly: (i) it activates later, so it has access to more information, (ii) it is used less frequently, so it can expend less effort, and (iii) each use of it has greater consequences, so it's more acceptable to just rely on forking to align incentives for this final layer. hybrid solutions there are also solutions that combine elements of the above techniques. some possible examples: time delays plus elected-specialist governance: this is one possible solution to the ancient conundrum of how to make an crypto-collateralized stablecoin whose locked funds can exceed the value of the profit-taking token without risking governance capture. the stable coin uses a price oracle constructed from the median of values submitted by n (eg. n = 13) elected providers. coin voting chooses the providers, but it can only cycle out one provider each week. if users notice that coin voting is bringing in untrustworthy price providers, they have n/2 weeks before the stablecoin breaks to switch to a different one. futarchy + anti-collusion = reputation: users vote with "reputation", a token that cannot be transferred. users gain more reputation if their decisions lead to desired results, and lose reputation if their decisions lead to undesired results. see here for an article advocating for a reputation-based scheme. loosely-coupled (advisory) coin votes: a coin vote does not directly implement a proposed change, instead it simply exists to make its outcome public, to build legitimacy for off-chain governance to implement that change. this can provide the benefits of coin votes, with fewer risks, as the legitimacy of a coin vote drops off automatically if evidence emerges that the coin vote was bribed or otherwise manipulated. but these are all only a few possible examples. there is much more that can be done in researching and developing non-coin-driven governance algorithms. the most important thing that can be done today is moving away from the idea that coin voting is the only legitimate form of governance decentralization. coin voting is attractive because it feels credibly neutral: anyone can go and get some units of the governance token on uniswap. in practice, however, coin voting may well only appear secure today precisely because of the imperfections in its neutrality (namely, large portions of the supply staying in the hands of a tightly-coordinated clique of insiders). we should stay very wary of the idea that current forms of coin voting are "safe defaults". there is still much that remains to be seen about how they function under conditions of more economic stress and mature ecosystems and financial markets, and the time is now to start simultaneously experimenting with alternatives. dark mode toggle what in the ethereum application ecosystem excites me 2022 dec 05 see all posts special thanks to matt huang, santi siri and tina zhen for feedback and review ten, five, or even two years ago, my opinions on what ethereum and blockchains can do for the world were very abstract. "this is a general-purpose technology, like c++", i would say; of course, it has specific properties like decentralization, openness and censorship resistance, but beyond that it's just too early to say which specific applications are going to make the most sense. today's world is no longer that world. today, enough time has passed that there are few ideas that are completely unexplored: if something succeeds, it will probably be some version of something that has already been discussed in blogs and forums and conferences on multiple occasions. we've also come closer to identifying fundamental limits of the space. many daos have had a fair chance with an enthusiastic audience willing to participate in them despite the inconveniences and fees, and many have underperformed. industrial supply-chain applications have not gone anywhere. decentralized amazon on the blockchain has not happened. but it's also a world where we have seen genuine and growing adoption of a few key applications that are meeting people's real needs and those are the applications that we need to focus on. hence my change in perspective: my excitement about ethereum is now no longer based in the potential for undiscovered unknowns, but rather in a few specific categories of applications that are proving themselves already, and are only getting stronger. what are these applications, and which applications am i no longer optimistic about? that is what this post will be about. 1. money: the first and still most important app when i first visited argentina in december last year, one of the experiences i remember well was walking around on christmas day, when almost everything is closed, looking for a coffee shop. after passing by about five closed ones, we finally found one that was open. when we walked in, the owner recognized me, and immediately showed me that he has eth and other crypto-assets on his binance account. we ordered tea and snacks, and we asked if we could pay in eth. the coffee shop owner obliged, and showed me the qr code for his binance deposit address, to which i sent about $20 of eth from my status wallet on my phone. this was far from the most meaningful use of cryptocurrency that is taking place in the country. others are using it to save money, transfer money internationally, make payments for large and important transactions, and much more. but even still, the fact that i randomly found a coffee shop and it happened to accept cryptocurrency showed the sheer reach of adoption. unlike wealthy countries like the united states, where financial transactions are easy to make and 8% inflation is considered extreme, in argentina and many other countries around the world, links to global financial systems are more limited and extreme inflation is a reality every day. cryptocurrency often steps in as a lifeline. in addition to binance, there is also an increasing number of local exchanges, and you can see advertisements for them everywhere including at airports. the one issue with my coffee transaction is that it did not really make pragmatic sense. the fee was high, about a third of the value of the transaction. the transaction took several minutes to confirm: i believe that at the time, status did not yet support sending proper eip-1559 transactions that more reliably confirm quickly. if, like many other argentinian crypto users, i had simply had a binance wallet, the transfer would have been free and instant. a year later, however, the calculus is different. as a side effect of the merge, transactions get included significantly more quickly and the chain has become more stable, making it safer to accept transactions after fewer confirmations. scaling technology such as optimistic and zk rollups is proceeding quickly. social recovery and multisig wallets are becoming more practical with account abstraction. these trends will take years to play out as the technology develops, but progress is already being made. at the same time, there is an important "push factor" driving interest in transacting on-chain: the ftx collapse, which has reminded everyone, latin americans included, that even the most trustworthy-seeming centralized services may not be trustworthy after all. cryptocurrency in wealthy countries in wealthy countries, the more extreme use cases around surviving high inflation and doing basic financial activities at all usually do not apply. but cryptocurrency still has significant value. as someone who has used it to make donations (to quite normal organizations in many countries), i can personally confirm that it is far more convenient than traditional banking. it's also valuable for industries and activities at risk of being deplatformed by payment processors a category which includes many industries that are perfectly legal under most countries' laws. there is also the important broader philosophical case for cryptocurrency as private money: the transition to a "cashless society" is being taken advantage of by many governments as an opportunity to introduce levels of financial surveillance that would be unimaginable 100 years ago. cryptocurrency is the only thing currently being developed that can realistically combine the benefits of digitalization with cash-like respect for personal privacy. but in either case, cryptocurrency is far from perfect. even with all the technical, user experience and account safety problems solved, it remains a fact that cryptocurrency is volatile, and the volatility can make it difficult to use for savings and business. for that reason, we have... stablecoins the value of stablecoins has been understood in the ethereum community for a long time. quoting a blog post from 2014: over the past eleven months, bitcoin holders have lost about 67% of their wealth and quite often the price moves up or down by as much as 25% in a single week. seeing this concern, there is a growing interest in a simple question: can we get the best of both worlds? can we have the full decentralization that a cryptographic payment network offers, but at the same time have a higher level of price stability, without such extreme upward and downward swings? and indeed, stablecoins are very popular among precisely those users who are making pragmatic use of cryptocurrency today. that said, there is a reality that is not congenial to cypherpunk values today: the stablecoins that are most successful today are the centralized ones, mostly usdc, usdt and busd. top cryptocurrency market caps, data from coingecko, 2022-11-30. three of the top six are centralized stablecoins. stablecoins issued on-chain have many convenient properties: they are open for use by anyone, they are resistant to the most large-scale and opaque forms of censorship (the issuer can blacklist and freeze addresses, but such blacklisting is transparent, and there are literal transaction fee costs associated with freezing each address), and they interact well with on-chain infrastructure (accounts, dexes, etc). but it's not clear how long this state of affairs will last, and so there is a need to keep working on other alternatives. i see the stablecoin design space as basically being split into three different categories: centralized stablecoins, dao-governed real-world-asset backed stablecoins and governance-minimized crypto-backed stablecoins. governance advantages disadvantages examples centralized stablecoins traditional legal entity • maximum efficiency • easy to understand vulnerable to risks of a single issuer and a single jurisdiction usdc, usdt, busd dao-governed rwa-backed stablecoins dao deciding on allowed collateral types and maximum per type • adds resilience by diversifying issuers and jurisdictions • still somewhat capital efficient vulnerable to repeated issuer fraud or coordinated takedown dai governance-minimized crypto-backed stablecoin price oracle only • maximum resilience • no outside dependencies • high collateral requirements • limited scale • sometimes needs negative interest rates rai, lusd from the user's perspective, the three types are arranged on a tradeoff spectrum between efficiency and resilience. usdc works today, and will almost certainly work tomorrow. but in the longer term, its ongoing stability depends on the macroeconomic and political stability of the united states, a continued us regulatory environment that supports making usdc available to everyone, and the trustworthiness of the issuing organization. rai, on the other hand, can survive all of these risks, but it has a negative interest rate: at the time of this writing, -6.7%. to make the system stable (so, not be vulnerable to collapse like luna), every holder of rai must be matched by a holder of negative rai (aka. a "borrower" or "cdp holder") who puts in eth as collateral. this rate could be improved with more people engaging in arbitrage, holding negative rai and balancing it out with positive usdc or even interest-bearing bank account deposits, but interest rates on rai will always be lower than in a functioning banking system, and the possibility of negative rates, and the user experience headaches that they imply, will always be there. the rai model is ultimately ideal for the more pessimistic lunarpunk world: it avoids all connection to non-crypto financial systems, making it much more difficult to attack. negative interest rates prevent it from being a convenient proxy for the dollar, but one way to adapt would be to embrace the disconnection: a governance-minimized stablecoin could track some non-currency asset like a global average cpi index, and advertise itself as representing abstract "best-effort price stability". this would also have lower inherent regulatory risk, as such an asset would not be attempting to provide a "digital dollar" (or euro, or...). dao-governed rwa-backed stablecoins, if they can be made to work well, could be a happy medium. such stablecoins could combine enough robustness, censorship resistance, scale and economic practicality to satisfy the needs of a large number of real-world crypto users. but making this work requires both real-world legal work to develop robust issuers, and a healthy dose of resilience-oriented dao governance engineering. in either case, any kind of stablecoin working well would be a boon for many kinds of currency and savings applications that are already concretely useful for millions of people today. 2. defi: keep it simple decentralized finance is, in my view, a category that started off honorable but limited, turned into somewhat of an overcapitalized monster that relied on unsustainable forms of yield farming, and is now in the early stages of setting down into a stable medium, improving security and refocusing on a few applications that are particularly valuable. decentralized stablecoins are, and probably forever will be, the most important defi product, but there are a few others that have an important niche: prediction markets: these have been a niche but stable pillar of decentralized finance since the launch of augur in 2015. since then, they have quietly been growing in adoption. prediction markets showed their value and their limitations in the 2020 us election, and this year in 2022, both crypto prediction markets like polymarket and play-money markets like metaculus are becoming more and more widely used. prediction markets are valuable as an epistemic tool, and there is a genuine benefit from using cryptocurrency in making these markets more trustworthy and more globally accessible. i expect prediction markets to not make extreme multibillion-dollar splashes, but continue to steadily grow and become more useful over time. other synthetic assets: the formula behind stablecoins can in principle be replicated to other real-world assets. interesting natural candidates include major stock indices and real estate. the latter will take longer to get right due to the inherent heterogeneity and complexity of the space, but it could be valuable for precisely the same reasons. the main question is whether or not someone can create the right balance of decentralization and efficiency that gives users access to these assets at reasonable rates of return. glue layers for efficiently trading between other assets: if there are assets on-chain that people want to use, including eth, centralized or decentralized stablecoins, more advanced synthetic assets, or whatever else, there will be value in a layer that makes it easy for users to trade between them. some users may want to hold usdc and pay transaction fees in usdc. others may hold some assets, but want to be able to instantly convert to pay someone who wants to be paid in another asset. there is also space for using one asset as collateral to take out loans of another asset, though such projects are most likely to succeed and avoid leading to tears if they keep leverage very limited (eg. not more than 2x). 3. the identity ecosystem: ens, siwe, poh, poaps, sbts "identity" is a complicated concept that can mean many things. some examples include: basic authentication: simply proving that action a (eg. sending a transaction or logging into a website) was authorized by some agent that has some identifier, such as an eth address or a public key, without attempting to say anything else about who or what the agent is. attestations: proving claims about an agent made by other agents ("bob attests that he knows alice", "the government of canada attests that charlie is a citizen") names: establishing consensus that a particular human-readable name can be used to refer to a particular agent. proof of personhood: proving that an agent is human, and guaranteeing that each human can only obtain one identity through the proof of personhood system (this is often done with attestations, so it's not an entirely separate category, but it's a hugely important special case) for a long time, i have been bullish on blockchain identity but bearish on blockchain identity platforms. the use cases mentioned above are really important to many blockchain use cases, and blockchains are valuable for identity applications because of their institution-independent nature and the interoperability benefits that they provide. but what will not work is an attempt to create a centralized platform to achieve all of these tasks from scratch. what more likely will work is an organic approach, with many projects working on specific tasks that are individually valuable, and adding more and more interoperability over time. and this is exactly what has happened since then. the sign in with ethereum (siwe) standard allows users to log into (traditional) websites in much the same way that you can use google or facebook accounts to log into websites today. this is actually useful: it allows you to interact with a site without giving google or facebook access to your private information or the ability to take over or lock you out of your account. techniques like social recovery could give users account recovery options in case they forget their password that are much better than what centralized corporations offer today. siwe is supported by many applications today, including blockscan chat, the end-to-end-encrypted email and notes service skiff, and various blockchain-based alternative social media projects. ens lets users have usernames: i have vitalik.eth. proof of humanity and other proof-of-personhood systems let users prove that they are unique humans, which is useful in many applications including airdrops and governance. poap (the "proof of attendance protocol", pronounced either "pope" or "poe-app" depending on whether you're a brave contrarian or a sheep) is a general-purpose protocol for issuing tokens that represent attestations: have you completed an educational course? have you attended an event? have you met a particular person? poaps could be used both as an ingredient in a proof-of-personhood protocol and as a way to try to determine whether or not someone is a member of a particular community (valuable for governance or airdrops). an nfc card that contains my ens name, and allows you to receive a poap verifying that you've met me. i'm not sure i want to create any further incentive for people to bug me really hard to get my poap, but this seems fun and useful for other people. each of these applications are useful individually. but what makes them truly powerful is how well they compose with each other. when i log on to blockscan chat, i sign in with ethereum. this means that i am immediately visible as vitalik.eth (my ens name) to anyone i chat with. in the future, to fight spam, blockscan chat could "verify" accounts by looking at on-chain activity or poaps. the lowest tier would simply be to verify that the account has sent or been the recipient in at least one on-chain transaction (as that requires paying fees). a higher level of verification could involve checking for balances of specific tokens, ownership of specific poaps, a proof-of-personhood profile, or a meta-aggregator like gitcoin passport. the network effects of these different services combine to create an ecosystem that provides some very powerful options for users and applications. an ethereum-based twitter alternative (eg. farcaster) could use poaps and other proofs of on-chain activity to create a "verification" feature that does not require conventional kyc, allowing anons to participate. such platforms could create rooms that are gated to members of a particular community or hybrid approaches where only community members can speak but anyone can listen. the equivalent of twitter polls could be limited to particular communities. equally importantly, there are much more pedestrian applications that are relevant to simply helping people make a living: verification through attestations can make it easier for people to prove that they are trustworthy to get rent, employment or loans. the big future challenge for this ecosystem is privacy. the status quo involves putting large amounts of information on-chain, which is something that is "fine until it's not", and eventually will become unpalatable if not outright risky to more and more people. there are ways to solve this problem by combining on-chain and off-chain information and making heavy use of zk-snarks, but this is something that will actually need to be worked on; projects like sismo and heyanon are an early start. scaling is also a challenge, but scaling can be solved generically with rollups and perhaps validiums. privacy cannot, and must be worked on intentionally for each application. 4. daos "dao" is a powerful term that captures many of the hopes and dreams that people have put into the crypto space to build more democratic, resilient and efficient forms of governance. it's also an incredibly broad term whose meaning has evolved a lot over the years. most generally, a dao is a smart contract that is meant to represent a structure of ownership or control over some asset or process. but this structure could be anything, from the lowly multisig to highly sophisticated multi-chamber governance mechanisms like those proposed for the optimism collective. many of these structures work, and many others cannot, or at least are very mismatched to the goals that they are trying to achieve. there are two questions to answer: what kinds of governance structures make sense, and for what use cases? does it make sense to implement those structures as a dao, or through regular incorporation and legal contracts? a particular subtlety is that the word "decentralized" is sometimes used to refer to both: a governance structure is decentralized if its decisions depend on decisions taken from a large group of participants, and an implementation of a governance structure is decentralized if it is built on a decentralized structure like a blockchain and is not dependent on any single nation-state legal system. decentralization for robustness one way to think about the distinction is: decentralized governance structure protects against attackers on the inside, and a decentralized implementation protects against powerful attackers on the outside ("censorship resistance"). first, some examples: higher need for protection from inside lower need for protection from inside higher need for protection from outside stablecoins the pirate bay, sci-hub lower need for protection from outside regulated financial institutions regular businesses the pirate bay and sci-hub are important case studies of something that is censorship-resistant, but does not need decentralization. sci-hub is largely run by one person, and if some part of sci-hub infrastructure gets taken down, she can simply move it somewhere else. the sci-hub url has changed many times over the years. the pirate bay is a hybrid: it relies on bittorrent, which is decentralized, but the pirate bay itself is a centralized convenience layer on top. the difference between these two examples and blockchain projects is that they do not attempt to protect their users against the platform itself. if sci-hub or the pirate bay wanted to harm their users, the worst they could do is either serve bad results or shut down either of which would only cause minor inconvenience until their users switch to other alternatives that would inevitably pop up in their absence. they could also publish user ip addresses, but even if they did that the total harm to users would still be much lower than, say, stealing all the users' funds. stablecoins are not like this. stablecoins are trying to create stable credibly neutral global commercial infrastructure, and this demands both lack of dependence on a single centralized actor on the outside and protection against attackers from the inside. if a stablecoin's governance is poorly designed, an attack on the governance could steal billions of dollars from users. at the time of this writing, makerdao has $7.8 billion in collateral, over 17x the market cap of the profit-taking token, mkr. hence, if governance was up to mkr holders with no safeguards, someone could buy up half the mkr, use that to manipulate the price oracles, and steal a large portion of the collateral for themselves. in fact, this actually happened with a smaller stablecoin! it hasn't happened to mkr yet largely because the mkr holdings are still fairly concentrated, with the majority of the mkr held by a fairly small group that would not be willing to sell because they believe in the project. this is a fine model to get a stablecoin started, but not a good one for the long term. hence, making decentralized stablecoins work long term requires innovating in decentralized governance that does not have these kinds of flaws. two possible directions include: some kind of non-financialized governance, or perhaps a bicameral hybrid where decisions need to be passed not just by token holders but also by some other class of user (eg. the optimism citizens' house or steth holders as in the lido two-chamber proposal) intentional friction, making it so that certain kinds of decisions can only take effect after a delay long enough that users can see that something is going wrong and escape the system. there are many subtleties in making governance that effectively optimizes for robustness. if the system's robustness depends on pathways that are only activated in extreme edge cases, the system may even want to intentionally test those pathways once in a while to make sure that they work much like the once-every-20-years rebuilding of ise jingu. this aspect of decentralization for robustness continues to require more careful thought and development. decentralization for efficiency decentralization for efficiency is a different school of thought: decentralized governance structure is valuable because it can incorporate opinions from more diverse voices at different scales, and decentralized implementation is valuable because it can sometimes be more efficient and lower cost than traditional legal-system-based approaches. this implies a different style of decentralization. governance decentralized for robustness emphasizes having a large number of decision-makers to ensure alignment with a pre-set goal, and intentionally makes pivoting more difficult. governance decentralized for efficiency preserves the ability to act rapidly and pivot if needed, but tries to move decisions away from the top to avoid the organization becoming a sclerotic bureaucracy. pod-based governance in ukraine dao. this style of governance improves efficiency by maximizing autonomy. decentralized implementations designed for robustness and decentralized implementations designed for efficiency are in one way similar: they both just involve putting assets into smart contracts. but decentralized implementations designed for efficiency are going to be much simpler: just a basic multisig will generally suffice. it's worth noting that "decentralizing for efficiency" is a weak argument for large-scale projects in the same wealthy country. but it's a stronger argument for very-small-scale projects, highly internationalized projects, and projects located in countries with inefficient institutions and weak rule of law. many applications of "decentralizing for efficiency" probably could also be done on a central-bank-run chain run by a stable large country; i suspect that both decentralized approaches and centralized approaches are good enough, and it's the path-dependent question of which one becomes viable first that will determine which approach dominates. decentralization for interoperability this is a fairly boring class of reasons to decentralize, but it's still important: it's easier and more secure for on-chain things to interact with other on-chain things, than with off-chain systems that would inevitably require an (attackable) bridge layer. if a large organization running on direct democracy holds 10,000 eth in its reserves, that would be a decentralized governance decision, but it would not be a decentralized implementation: in practice, that country would have a few people managing the keys and that storage system could get attacked. there is also a governance angle to this: if a system provides services to other daos that are not capable of rapid change, it is better for that system to itself be incapable of rapid change, to avoid "rigidity mismatch" where a system's dependencies break and that system's rigidity renders it unable to adapt to the break. these three "theories of decentralization" can be put into a chart as follows: why decentralize governance structure why decentralize implementation decentralization for robustness defense against inside threats (eg. sbf) defense against outside threats, and censorship resistance decentralization for efficiency greater efficiency from accepting input from more voices and giving room for autonomy smart contracts often more convenient than legal systems decentralization for interoperability to be rigid enough to be safe to use by other rigid systems to more easily interact with other decentralized things decentralization and fancy new governance mechanisms over the last few decades, we've seen the development of a number of fancy new governance mechanisms: quadratic voting futarchy liquid democracy decentralized conversation tools like pol.is these ideas are an important part of the dao story, and they can be valuable for both robustness and efficiency. the case for quadratic voting relies on a mathematical argument that it makes the exactly correct tradeoff between giving space for stronger preferences to outcompete weaker but more popular preferences and not weighting stronger preferences (or wealthy actors) too much. but people who have used it have found that it can improve robustness too. newer ideas, like pairwise matching, intentionally sacrifice mathematically provable optimality for robustness in situations where the mathematical model's assumptions break. these ideas, in addition to more "traditional" centuries-old ideas around multicameral architectures and intentional indirection and delays, are going to be an important part of the story in making daos more effective, though they will also find value in improving the efficiency of traditional organizations. case study: gitcoin grants we can analyze the different styles of decentralization through an interesting edge-case: gitcoin grants. should gitcoin grants be an on-chain dao, or should it just be a centralized org? here are some possible arguments for gitcoin grants to be a dao: it holds and deals with cryptocurrency, because most of its users and funders are ethereum users secure quadratic funding is best done on-chain (see next section on blockchain voting, and implementation of on-chain qf here), so you reduce security risks if the result of the vote feeds into the system directly it deals with communities all around the world, and so benefits from being credibly neutral and not centered around a single country. it benefits from being able to give its users confidence that it will still be around in five years, so that public goods funders can start projects now and hope to be rewarded later. these arguments lean toward decentralization for robustness and decentralization for interoperability of the superstructure, though the individual quadratic funding rounds are more in the "decentralization for efficiency" school of thought (the theory behind gitcoin grants is that quadratic funding is a more efficient way to fund public goods). if the robustness and interoperability arguments did not apply, then it probably would have been better to simply run gitcoin grants as a regular company. but they do apply, and so gitcoin grants being a dao makes sense. there are plenty of other examples of this kind of argument applying, both for daos that people increasingly rely on for their day-to-day lives, and for "meta-daos" that provide services to other daos: proof of humanity kleros chainlink stablecoins blockchain layer 2 protocol governance i don't know enough about all of these systems to testify that they all do optimize for decentralization-for-robustness enough to satisfy my standards, but hopefully it should be obvious by now that they should. the main thing that does not work well are daos that require pivoting ability that is in conflict with robustness, and that do not have a sufficient case to "decentralize for efficiency". large-scale companies that mainly interface with us users would be one example. when making a dao, the first thing is to determine whether or not it is worth it to structure the project as a dao, and the second thing is to determine whether it's targeting robustness or efficiency: if the former, deep thought into governance design is also required, and if the latter, then either it's innovating on governance via mechanisms like quadratic funding, or it should just be a multisig. 5. hybrid applications there are many applications that are not entirely on-chain, but that take advantage of both blockchains and other systems to improve their trust models. voting is an excellent example. high assurances of censorship resistance, auditability and privacy are all required, and systems like maci effectively combine blockchains, zk-snarks and a limited centralized (or m-of-n) layer for scalability and coercion resistance to achieve all of these guarantees. votes are published to the blockchain, so users have a way independent of the voting system to ensure that their votes get included. but votes are encrypted, preserving privacy, and a zk-snark-based solution is used to ensure that the final result is the correct computation of the votes. diagram of how maci works, combining together blockchains for censorship resistance, encryption for privacy, and zk-snarks to ensure the result is correct without compromising on the other goals. voting in existing national elections is already a high-assurance process, and it will take a long time before countries and citizens are comfortable with the security assurances of any electronic ways to vote, blockchain or otherwise. but technology like this can be valuable very soon in two other places: increasing the assurance of voting processes that already happen electronically today (eg. social media votes, polls, petitions) creating new forms of voting that allow citizens or members of groups to give rapid feedback, and baking high assurance into those from the start going beyond voting, there is an entire field of potential "auditable centralized services" that could be well-served by some form of hybrid off-chain validium architecture. the easiest example of this is proof of solvency for exchanges, but there are plenty of other possible examples: government registries corporate accounting games (see dark forest for an example) supply chain applications tracking access authorization ... as we go further down the list, we get to use cases that are lower and lower value, but it is important to remember that these use cases are also quite low cost. validiums do not require publishing everything on-chain. rather, they can be simple wrappers around existing pieces of software that maintain a merkle root (or other commitment) of the database and occasionally publish the root on-chain along with a snark proving that it was updated correctly. this is a strict improvement over existing systems, because it opens the door for cross-institutional proofs and public auditing. so how do we get there? many of these applications are being built today, though many of these applications are seeing only limited usage because of the limitations of present-day technology. blockchains are not scalable, transactions until recently took a fairly long time to reliably get included on the chain, and present-day wallets give users an uncomfortable choice between low convenience and low security. in the longer term, many of these applications will need to overcome the specter of privacy issues. these are all problems that can be solved, and there is a strong drive to solve them. the ftx collapse has shown many people the importance of truly decentralized solutions to holding funds, and the rise of erc-4337 and account abstraction wallets gives us an opportunity to create such alternatives. rollup technology is rapidly progressing to solve scalability, and transactions already get included much more quickly on-chain than they did three years ago. but what is also important is to be intentional about the application ecosystem itself. many of the more stable and boring applications do not get built because there is less excitement and less short-term profit to be earned around them: the luna market cap got to over $30 billion, while stablecoins striving for robustness and simplicity often get largely ignored for years. non-financial applications often have no hope of earning $30 billion because they do not have a token at all. but it is these applications that will be most valuable for the ecosystem in the long term, and that will bring the most lasting value to both their users and those who build and support them. dark mode toggle the end of my childhood 2024 jan 31 see all posts one of my most striking memories from my last two years was speaking at hackathons, visiting hacker houses, and doing zuzalu in montenegro, and seeing people a full decade younger than myself taking on leading roles, as organizers or as developers, in all kinds of projects: crypto auditing, ethereum layer 2 scaling, synthetic biology and more. one of the memes of the core organizing team at zuzalu was the 21-year-old nicole sun, and a year earlier she had invited me to visit a hacker house in south korea: a ~30-person gathering where, for the first time that i can recall, i was by a significant margin the oldest person in the room. when i was as old as those hacker house residents are now, i remember lots of people lavishing me with praise for being one of these fancy young wunderkinds transforming the world like zuckerberg and so on. i winced at this somewhat, both because i did not enjoy that kind of attention and because i did not understand why people had to translate "wonder kid" into german when it works perfectly fine in english. but watching all of these people go further than i did, younger than i did, made me clearly realize that if that was ever my role, it is no longer. i am now in some different kind of role, and it is time for the next generation to take up the mantle that used to be mine. the path leading up to the hacker house in seoul, august 2022. photo'd because i couldn't tell which house i was supposed to be entering and i was communicating with the organizers to get that information. of course, the house ended up not being on this path at all, but rather in a much more visible venue about twenty meters to the right of it. 1 as a proponent of life extension (meaning, doing the medical research to ensure that humans can literally live thousands or millions of years), people often ask me: isn't the meaning of life closely tied to the fact that it's finite: you only have a small bit, so you have to enjoy it? historically, my instinct has been to dismiss this idea: while is it true as a matter of psychology that we tend to value things more if they are limited or scarce, it's simply absurd to argue that the ennui of a great prolonged existence could be so bad that it's worse than literally no longer existing. besides, i would sometimes think, even if eternal life proved to be that bad, we could always simultaneously dial up our "excitement" and dial down our longevity by simply choosing to hold more wars. the fact that the non-sociopathic among us reject that option today strongly suggests to me that we would reject it for biological death and suffering as well, as soon as it becomes a practical option to do so. as i have gained more years, however, i realized that i do not even need to argue any of this. regardless of whether our lives as a whole are finite or infinite, every single beautiful thing in our lives is finite. friendships that you thought are forever turn out to slowly fade away into the mists of time. your personality can completely change in 10 years. cities can transform completely, for better or sometimes for worse. you may move to a new city yourself, and restart the process of getting acquainted with your physical environment from scratch. political ideologies are finite: you may build up an entire identity around your views on top marginal tax rates and public health care, and ten years later feel completely lost once people seem to completely stop caring about those topics and switch over to spending their whole time talking about "wokeness", the "bronze age mindset" and "e/acc". a person's identity is always tied to their role in the broader world that they are operating in, and over a decade, not only does a person change, but so does the world around them. one change in my thinking that i have written about before is how my thinking involves less economics than it did ten years ago. the main cause of this shift is that i spent a significant part of the first five years of my crypto life trying to invent the mathematically provably optimal governance mechanism, and eventually i discovered some fundamental impossibility results that made it clear to me that (i) what i was looking for was impossible, and (ii) the most important variables that make the difference between existing flawed systems succeeding or failing in practice (often, the degree of coordination between subgroups of participants, but also other things that we often black-box as "culture") are variables that i was not even modeling. before, mathematics was a primary part of my identity: i was heavily involved in math competitions in high school, and soon after i got into crypto, i began doing a lot of coding, in ethereum, bitcoin and elsewhere, i was getting excited about every new cryptography protocol, and economics too seemed to me to be part of that broader worldview: it's the mathematical tool for understanding and figuring out how to improve the social world. all the pieces neatly fit together. now, those pieces fit together somewhat less. i do still use mathematics to analyze social mechanisms, though the goal is more often to come up with rough first-pass guesses about what might work and mitigate worst-case behavior (which, in a real-world setting, would be usually done by bots and not humans) rather than explain average-case behavior. now, much more of my writing and thinking, even when supporting the same kinds of ideals that i supported a decade ago, often uses very different kinds of arguments. one thing that fascinates me about modern ai is that it lets us mathematically and philosophically engage with the hidden variables guiding human interaction in a different way: ai can make "vibes" legible. all of these deaths, births and rebirths, whether of ideas or collections of people, are ways in which life is finite. these deaths and births would continue to take place in a world where we lived two centuries, a millennium, or the same lifetime as a main-sequence star. and if you personally feel like life doesn't have enough finiteness and death and rebirth in it, you don't have to start wars to add more: you can also just make the same choice that i did and become a digital nomad. 2 "grads are falling in mariupol". i still remember anxiously watching the computer screen in my hotel room in denver, on february 23, 2022, at 7:20 pm local time. for the past two hours, i had been simultaneously scrolling twitter for updates and repeatedly pinging my dad, who has having the very same thoughts and fears that i was, until he finally sent me that fateful reply. i sent out a tweet making my position on the issue as clear as possible and i kept watching. i stayed up very late that night. the next morning i woke up to the ukraine government twitter account desperately asking for donations in cryptocurrency. at first, i thought that there is no way this could be real, and i became very worried that the account was opportunistically hacked: someone, perhaps the russian government itself, taking advantage of everyone's confusion and desperation to steal some money. my "security mindset" instinct took over, and i immediately started tweeting to warn people to be careful, all while going through my network to find people who could confirm or deny if the eth address is genuine. an hour later, i was convinced that it was in fact genuine, and i publicly relayed my conclusion. and about an hour after that, a family member sent me a message pointing out that, given what i had already done, it would be better for my safety for me to not go back to russia again. eight months later, i was watching the crypto world go through a convulsion of a very different sort: the extremely public demise of sam bankman-fried and ftx. at the time, someone posted on twitter a long list of "crypto main characters", showing which ones had fallen and which ones were still intact. the casualty rate was massive: the sbf situation was not unique: it mix-and-matched aspects of mtgox and several other convulsions that had engulfed the crypto space before. but it was a moment where i realized, all at once, that most of the people i had looked up to as guiding lights of the crypto space that i could comfortably follow in the footsteps of back in 2014 were no more. people looking at me from afar often think of me as a high-agency person, presumably because this is what you would expect of a "main character" or a "project founder" who "dropped out of college". in reality, however, i was anything but. the virtue i valorized as a kid was not the virtue of creativity in starting a unique new project, or the virtue of showing bravery in a once-in-a-generation moment that calls for it, but rather the virtue of being a good student who shows up on time, does his homework and gets a 99 percent average. my decision to drop out of college was not some kind of big brave step done out of conviction. it started with me in early 2013 deciding to take a co-op term in the summer to work for ripple. when us visa complications prevented that, i instead spent the summer working with my bitcoin magazine boss and friend mihai alisie in spain. near the end of august, i decided that i needed to spend more time exploring the crypto world, and so i extended my vacation to 12 months. only in january 2014, when i saw the social proof of hundreds of people cheering on my presentation introducing ethereum at btc miami, did i finally realize that the choice was made for me to leave university for good. most of my decisions in ethereum involved responding to other people's pressures and requests. when i met vladimir putin in 2017, i did not try to arrange the meeting; rather, someone else suggested it, and i pretty much said "ok sure". now, five years later, i finally realized that (i) i had been complicit in legitimizing a genocidal dictator, and (ii) within the crypto space too, i no longer had the luxury of sitting back and letting mystical "other people" run the show. these two events, as different as they are in the type and the scale of their tragedy, both burned into my mind a similar lesson: that i actually have responsibilities in this world, and i need to be intentional about how i operate. doing nothing, or living on autopilot and letting myself simply become part of the plans of others, is not an automatically safe, or even blameless, course of action. i was one of the mystical other people, and it was up to me to play the part. if i do not, and the crypto space either stagnates or becomes dominated by opportunistic money-grabbers more than it otherwise would have as a result, i have only myself to blame. and so i decided to become careful in which of others' plans i go along with, and more high-agency in what plans i craft myself: fewer ill-conceived meetings with random powerful people who were only interested in me as a source of legitimacy, and more things like zuzalu. the zuzalu flags in montenegro, spring 2023. 3 on to happier things or at least, things that are challenging in the way that a math puzzle is challenging, rather than challenging in the way that falling down in the middle of a run and needing to walk 2km with a bleeding knee to get medical attention is challenging (no, i won't share more details; the internet has already proven top notch at converting a photo of me with a rolled-up usb cable in my pocket into an internet meme insinuating something completely different, and i certainly do not want to give those characters any more ammunition). i have talked before about the changing role of economics, the need to think differently about motivation (and coordination: we are social creatures, so the two are in fact intimately linked), and the idea that the world is becoming a "dense jungle": big government, big business, big mob, and big x for pretty much any x will all continue to grow, and they will have more and more frequent and complicated interactions with each other. what i have not yet talked as much about is how many of these changes affect the crypto space itself. the crypto space was born in late 2008, in the aftermath of the global financial crisis. the genesis block of the bitcoin blockchain contained a reference to this famous article from the uk's the times: the early memes of bitcoin were heavily influenced by these themes. bitcoin is there to abolish the banks, which is a good thing to do because the banks are unsustainable megaliths that keep creating financial crises. bitcoin is there to abolish fiat currency, because the banking system can't exist without the underlying central banks and the fiat currencies that they issue and furthermore, fiat currency enables money printing which can fund wars. but iin the fifteen years since then, the broader public discourse as a whole seems to have to a large extent moved beyond caring about money and banks. what is considered important now? well, we can ask the copy of mixtral 8x7b running on my new gpu laptop: once again, ai can make vibes legible. no mention of money and banks or government control of currency. trade and inequality are listed as concerns globally, but from what i can tell, the problems and solutions being discussed are more in the physical world than the digital world. is the original "story" of crypto falling further and further behind the times? there are two sensible responses to this conundrum, and i believe that our ecosystem would benefit from embracing both of them: remind people that money and finance still do matter, and do a good job of serving the world's underserved in that niche extend beyond finance, and use our technology to build a more holistic vision of an alternative, more free and open and democratic tech stack, and how that could build toward either a broadly better society, or at least tools to help those who are excluded from mainstream digital infrastructure today. is important, and i would argue that the crypto space is uniquely positioned to provide value there. crypto is one of the few tech industries that is genuinely highly decentralized, with developers spread out all over the globe: source: electric capital's 2023 crypto developer report having visited many of the new global hubs of crypto over the past year, i can confirm that this is the case. more and more of the largest crypto projects are headquartered in all kinds of far-flung places around the world, or even nowhere. furthermore, non-western developers often have a unique advantage in understanding the concrete needs of crypto users in low-income countries, and being able to create products that satisfy those needs. when i talk to many people from san francisco, i get a distinct impression that they think that ai is the only thing that matters, san francisco is the capital of ai, and therefore san francisco is the only place that matters. "so, vitalik, why are you not settled down in the bay with an o1 visa yet"? crypto does not need to play this game: it's a big world, and it only takes one visit to argentina or turkey or zambia to remind ourselves that many people still do have important problems that have to do with access to money and finance, and there is still an opportunity to do the complicated work of balancing user experience and decentralization to actually solve those problems in a sustainable way. the other vision is the one that i outlined in my recent post, "make ethereum cypherpunk again". rather than just focusing on money, or being an "internet of value", i argued that the ethereum community should expand its horizons. we should create an entire decentralized tech stack a stack that is independent from the traditional silicon valley tech stack to the same extent that eg. the chinese tech stack is and compete with centralized tech companies at every level. reproducing that table here: traditional stack decentralized stack banking system eth, stablecoins, l2s for payments, dexes (note: still need banks for loans) receipts links to transactions on block explorers corporations daos dns (.com, .io, etc) ens (.eth) regular email encrypted email (eg. skiff) regular messaging (eg. telegram) decentralized messaging (eg. status) sign in with google, twitter, wechat sign in with ethereum, zupass, attestations via eas, poaps, zu-stamps... + social recovery publishing blogs on medium, etc publishing self-hosted blogs on ipfs (eg. using fleek) twitter, facebook lens, farcaster... limit bad actors through all-seeing big brother constrain bad actors through zero knowledge proofs after i made that post, some readers reminded me that a major missing piece from this stack is democratic governance technology: tools for people to collectively make decisions. this is something that centralized tech does not really even try to provide, because the assumption that each indidvidual company is just run by a ceo, and oversight is provided by... err... a board. ethereum has benefited from very primitive forms of democratic governance technology in the past already: when a series of contentious decisions, such as the dao fork and several rounds of issuance decrease, were made in 2016-2017, a team from shanghai made a platform called carbonvote, where eth holders could vote on decisions. the eth vote on the dao fork. the votes were advisory in nature: there was no hard agreement that the results would determine what happens. but they helped give core developers the confidence to actually implement a series of eips, knowing that the mass of the community would be behind them. today, we have access to proofs of community membership that are much richer than token holdings: poaps, gitcoin passport scores, zu stamps, etc. from these things all together, we can start to see the second vision for how the crypto space can evolve to better meet the concerns and needs of the 21st century: create a more holistic trustworthy, democratic, and decentralized tech stack. zero knowledge proofs are key here in expanding the scope of what such a stack can offer: we can get beyond the false binary of "anonymous and therefore untrusted" vs "verified and kyc'd", and prove much more fine-grained statements about who we are and what permissions we have. this allows us to resolve concerns around authenticity and manipulation guarding against "the big brother outside" and concerns around privacy guarding against "the big brother within" at the same time. this way, crypto is not just a finance story, it can be part of a much broader story of making a better type of technology. 4 but how, beyond telling stories do we make this happen? here, we get back to some of the issues that i raised in my post from three years ago: the changing nature of motivation. often, people with an overly finance-focused theory of motivation or at least, a theory of motivation within which financial motives can be understood and analyzed and everything else is treated as that mysterious black box we call "culture" are confused by the space because a lot of the behavior seems to go against financial motives. "users don't care about decentralization", and yet projects still often try hard to decentralize. "consensus runs on game theory", and yet successful social campaigns to chase people off the dominant mining or staking pool have worked in bitcoin and in ethereum. it occurred to me recently that no one that i have seen has attempted to create a basic functional map of the crypto space working "as intended", that tries to include more of these actors and motivations. so let me quickly make an attempt now: this map itself is an intentional 50/50 mix of idealism and "describing reality". it's intended to show four major constituencies of the ecosystem that can have a supportive and symbiotic relationship with each other. many crypto institutions in practice are a mix of all four. each of the four parts has something key to offer to the machine as a whole: token holders and defi users contribute greatly to financing the whole thing, which has been key to getting technologies like consensus algorithms and zero-knowledge proofs to production quality. intellectuals provide the ideas to make sure that the space is actually doing something meaningful. builders bridge the gap and try to build applications that serve users and put the ideas into practice. pragmatic users are the people we are ultimately serving. and each of the four groups has complicated motivations, which interplay with the other groups in all kinds of complicated ways. there are also versions of each of these four that i would call "malfunctioning": apps can be extractive, defi users can unwittingly entrench extractive apps' network effects, pragmatic users can entrench centralized workflows, and intellectuals can get overly worked up on theory and overly focus on trying to solve all problems by yelling at people for being "misaligned" without appreciating that the financial side of motivation (and the "user inconvenience" side of demotivation) matters too, and can and should be fixed. often, these groups have a tendency to scoff at each other, and at times in my history i have certainly played a part in this. some blockchain projects openly try to cast off the idealism that they see as naive, utopian and distracting, and focus directly on applications and usage. some developers disparage their token holders, and their dirty love of making money. still other developers disparage the pragmatic users, and their dirty willingness to use centralized solutions when those are more convenient for them. but i think there is an opportunity to improve understanding between the four groups, where each side understands that it is ultimately dependent on the other three, works to limit its own excesses, and appreciates that in many cases their dreams are less far apart than they think. this is a form of peace that i think is actually possible to achieve, both within the "crypto space", and between it and adjacent communities whose values are highly aligned. 5 one of the beautiful things about crypto's global nature is the window that it has given me to all kinds of fascinating cultures and subcultures around the world, and how they interact with the crypto universe. i still remember visiting china for the first time in 2014, and seeing all the signs of brightness and hope: exchanges scaling up to hundreds of employees even faster than those in the us, massive-scale gpu and later asic farms, and projects with millions of users. silicon valley and europe, meanwhile, have for a long time been key engines of idealism in the space, in their two distinct flavors. ethereum's development was, almost since the beginning, de-facto headquartered in berlin, and it was out of european open-source culture that a lot of the early ideas for how ethereum could be used in non-financial applications emerged. a diagram of ethereum and two proposed non-blockchain sister protocols whisper and swarm, which gavin wood used in many of his early presentations. silicon valley (by which, of course, i mean the entire san francisco bay area), was another hotbed of early crypto interest, mixed in with various ideologies such as rationalism, effective altruism and transhumanism. in the 2010s these ideas were all new, and they felt "crypto-adjacent": many of the people who were interested in them, were also interested in crypto, and likewise in the other direction. elsewhere, getting regular businesses to use cryptocurrency for payments was a hot topic. in all kind of places in the world, one would find people accepting bitcoin, including even japanese waiters taking bitcoin for tips: since then, these communities have experienced a lot of change. china saw multiple crypto crackdowns, in addition to other broader challenges, leading to singapore becoming a new home for many developers. silicon valley splintered internally: rationalists and ai developers, basically different wings of the same team back as recently as 2020 when scott alexander was doxxed by the new york times, have since become separate and dueling factions over the question of optimism vs pessimism about the default path of ai. the regional makeup of ethereum changed significantly, especially during the 2018-era introduction of totally new teams to work on proof of stake, though more through addition of the new than through demise of the old. death, birth and rebirth. there are many other communities that are worth mentioning. when i first visited taiwan many times in 2016 and 2017, what struck me most was the combination of capacity for self-organization and willingness to learn of the people there. whenever i would write a document or blog post, i would often find that within a day a study club would independently form and start excitedly annotating every paragraph of the post on google docs. more recently, members of the taiwanese ministry of digital affairs took a similar excitement to glen weyl's ideas of digital democracy and "plurality", and soon posted an entire mind map of the space (which includes a lot of ethereum applications) on their twitter account. paul graham has written about how every city sends a message: in new york, "you should make more money". in boston, you really should get around to reading all those books". in silicon valley, "you should be more powerful". when i visit taipei, the message that comes to my mind is "you should rediscover your inner high school student". glen weyl and audrey tang presenting at a study session at the nowhere book shop in taipei, where i had presented on community notes four months earlier when i visited argentina several times over the past few years, i was struck by the hunger and willingness to build and apply the technologies and ideas that ethereum and the broader cryptoverse have to offer. if places like siilicon valley are frontiers, filled with abstract far-mode thinking about a better future, places like argentina are frontlines, filled with an active drive to meet challenges that need to be handled today: in argentina's case, ultra-high inflation and limited connection to global financial systems. the amount of crypto adoption there is off the charts: i get recognized in the street more frequently in buenos aires than in san francisco. and there are many local builders, with a surprisingly healthy mix of pragmatism and idealism, working to meet people's challenges, whether it's crypto/fiat conversion or improving the state of ethereum nodes in latin america. myself and friends in a coffee shop in buenos aires, where we paid in eth. there are far too many others to properly mention: the cosmopolitanism and highly international crypto communities based in dubai, the growing zk community everywhere in east and southeast asia, the energetic and pragmatic builders in kenya, the public-goods-oriented solarpunk communities of colorado, and more. and finally, zuzalu in 2023 ended up creating a beautiful floating sub-community of a very different kind, which will hopefully flourish on its own in the years to come. this is a significant part of what attracts me about the network states movement at its best: the idea that cultures and communities are not just something to be defended and preserved, but also something that can be actively created and grown. 6 there are many lessons that one learns when growing up, and the lessons are different for different people. for me, a few are: greed is not the only form of selfishness. lots of harm can come from cowardice, laziness, resentment, and many other motives. furthermore, greed itself can come in many forms: greed for social status can often be just as harmful as greed for money or power. as someone raised in my gentle canadian upbringing, this was a major update: i felt like i had been taught to believe that greed for money and power is the root of most evils, and if i made sure i was not greedy for those things (eg. by repeatedly fighting to reduce the portion of the eth supply premine that went to the top-5 "founders") i satisfied my responsibility to be a good person. this is of course not true. you're allowed to have preferences without needing to have a complicated scientific explanation of why your preferences are the true absolute good. i generally like utilitarianism and find it often unfairly maligned and wrongly equated with cold-heartedness, but this is one place where i think ideas like utilitarianism in excess can sometimes lead human beings astray: there's a limit to how much you can change your preferences, and so if you push too hard, you end up inventing reasons for why every single thing you prefer is actually objectively best at serving general human flourishing. this often leads you to try to convince others that these back-fitted arguments are correct, leading to unneeded conflict. a related lesson is that a person can be a bad fit for you (for any context: work, friendship or otherwise) without being a bad person in some absolute sense. the importance of habits. i intentionally keep many of my day-to-day personal goals limited. for example, i try to do one 20-kilometer run a month, and "whatever i can" beyond that. this is because the only effective habits are the habits that you actually keep. if something is too difficult to maintain, you will give up on it. as a digital nomad who regularly jumps continents and makes dozens of flights per year, routine of any kind is difficult for me, and i have to work around that reality. though duolingo's gamification, pushing you to maintain a "streak" by doing at least something every day, actually does work on me. making active decisions is hard, and so it's always best to make active decisions that make the most long-term impact on your mind, by reprogramming your mind to default into a different pattern. there is a long tail of these that each person learns, and in principle i could go for longer. but there's also a limit to how much it's actually possible to learn from simply reading other people's experiences. as the world starts to change at a more rapid pace, the lessons that are available from other people's accounts also become outdated at a more rapid pace. so to a large extent, there is also no substitute for simply doing things the slow way and gaining personal experience. 7 every beautiful thing in the social world a community, an ideology, a "scene", or a country, or at the very small scale a company, a family or a relationship was created by people. even in those few cases where you could write a plausible story about how it existed since the dawn of human civilization and the eighteen tribes, someone at some point in the past had to actually write that story. these things are finite both the thing in itself, as a part of the world, and the thing as you experience it, an amalgamation of the underlying reality and your own ways of conceiving and interpreting it. and as communities, places, scenes, companies and families fade away, new ones have to be created to replace them. for me, 2023 has been a year of watching many things, large and small, fade into the distance of time. the world is rapidly changing, the frameworks i am forced to use to try to make sense of the world are changing, and the role i play in affecting the world is changing. there is death, a truly inevitable type of death that will continue to be with us even after the blight of human biological aging and mortality is purged from our civilization, but there is also birth and rebirth. and continuing to stay active and doing what we can to create the new is a task for each one of us. dark mode toggle what do i think about biometric proof of personhood? 2023 jul 24 see all posts special thanks to the worldcoin team, the proof of humanity community and andrew miller for discussion. one of the trickier, but potentially one of the most valuable, gadgets that people in the ethereum community have been trying to build is a decentralized proof-of-personhood solution. proof of personhood, aka the "unique-human problem", is a limited form of real-world identity that asserts that a given registered account is controlled by a real person (and a different real person from every other registered account), ideally without revealing which real person it is. there have been a few efforts at tackling this problem: proof of humanity, brightid, idena and circles come up as examples. some of them come with their own applications (often a ubi token), and some have found use in gitcoin passport to verify which accounts are valid for quadratic voting. zero-knowledge tech like sismo adds privacy to many of these solutions. more recently, we have seen the rise of a much larger and more ambitious proof-of-personhood project: worldcoin. worldcoin was co-founded by sam altman, who is best known for being the ceo of openai. the philosophy behind the project is simple: ai is going to create a lot of abundance and wealth for humanity, but it also may kill very many people's jobs and make it almost impossible to tell who even is a human and not a bot, and so we need to plug that hole by (i) creating a really good proof-of-personhood system so that humans can prove that they actually are humans, and (ii) giving everyone a ubi. worldcoin is unique in that it relies on highly sophisticated biometrics, scanning each user's iris using a piece of specialized hardware called "the orb": the goal is to produce a large number of these orbs and widely distribute them around the world and put them in public places to make it easy for anyone to get an id. to worldcoin's credit, they have also committed to decentralize over time. at first, this means technical decentralization: being an l2 on ethereum using the optimism stack, and protecting users' privacy with zk-snarks and other cryptographic techniques. later on, it includes decentralizing governance of the system itself. worldcoin has been criticized for privacy and security concerns around the orb, design issues in its "coin", and for ethical issues around some choices that the company has made. some of the criticisms are highly specific, focusing on decisions made by the project that could easily have been made in another way and indeed, that the worldcoin project itself may be willing to change. others, however, raise the more fundamental concern of whether or not biometrics not just the eye-scanning biometrics of worldcoin, but also the simpler face-video-uploads and verification games used in proof of humanity and idena are a good idea at all. and still others criticize proof of personhood in general. risks include unavoidable privacy leaks, further erosion of people's ability to navigate the internet anonymously, coercion by authoritarian governments, and the potential impossibility of being secure at the same time as being decentralized. this post will talk about these issues, and go through some arguments that can help you decide whether or not bowing down and scanning your eyes (or face, or voice, or...) before our new spherical overlords is a good idea, and whether or not the natural alternatives either using social-graph-based proof of personhood or giving up on proof of personhood entirely are any better. what is proof of personhood and why is it important? the simplest way to define a proof-of-personhood system is: it creates a list of public keys where the system guarantees that each key is controlled by a unique human. in other words, if you're a human, you can put one key on the list, but you can't put two keys on the list, and if you're a bot you can't put any keys on the list. proof of personhood is valuable because it solves a lot of anti-spam and anti-concentration-of-power problems that many people have, in a way that avoids dependence on centralized authorities and reveals the minimal information possible. if proof of personhood is not solved, decentralized governance (including "micro-governance" like votes on social media posts) becomes much easier to capture by very wealthy actors, including hostile governments. many services would only be able to prevent denial-of-service attacks by setting a price for access, and sometimes a price high enough to keep out attackers is also too high for many lower-income legitimate users. many major applications in the world today deal with this issue by using government-backed identity systems such as credit cards and passports. this solves the problem, but it makes large and perhaps unacceptable sacrifices on privacy, and can be trivially attacked by governments themselves. how many proof of personhood proponents see the two-sided risk that we are facing. image source. in many proof-of-personhood projects not just worldcoin, but also proof of humanity, circles and others the "flagship application" is a built-in "n-per-person token" (sometimes called a "ubi token"). each user registered in the system receives some fixed quantity of tokens each day (or hour, or week). but there are plenty of other applications: airdrops for token distributions token or nft sales that give more favorable terms to less-wealthy users voting in daos a way to "seed" graph-based reputation systems quadratic voting (and funding, and attention payments) protection against bots / sybil attacks in social media an alternative to captchas for preventing dos attacks in many of these cases, the common thread is a desire to create mechanisms that are open and democratic, avoiding both centralized control by a project's operators and domination by its wealthiest users. the latter is especially important in decentralized governance. in many of these cases, existing solutions today rely on some combination of (i) highly opaque ai algorithms that leave lots of room to undetectably discriminate against users that the operators simply do not like, and (ii) centralized ids, aka "kyc". an effective proof-of-personhood solution would be a much better alternative, achieving the security properties that those applications need without the pitfalls of the existing centralized approaches. what are some early attempts at proof of personhood? there are two main forms of proof of personhood: social-graph-based and biometric. social-graph based proof of personhood relies on some form of vouching: if alice, bob, charlie and david are all verified humans, and they all say that emily is a verified human, then emily is probably also a verified human. vouching is often enhanced with incentives: if alice says that emily is a human, but it turns out that she is not, then alice and emily may both get penalized. biometric proof of personhood involves verifying some physical or behavioral trait of emily, that distinguishes humans from bots (and individual humans from each other). most projects use a combination of the two techniques. the four systems i mentioned at the beginning of the post work roughly as follows: proof of humanity: you upload a video of yourself, and provide a deposit. to be approved, an existing user needs to vouch for you, and an amount of time needs to pass during which you can be challenged. if there is a challenge, a kleros decentralized court determines whether or not your video was genuine; if it is not, you lose your deposit and the challenger gets a reward. brightid: you join a video call "verification party" with other users, where everyone verifies each other. higher levels of verification are available via bitu, a system in which you can get verified if enough other bitu-verified users vouch for you. idena: you play a captcha game at a specific point in time (to prevent people from participating multiple times); part of the captcha game involves creating and verifying captchas that will then be used to verify others. circles: an existing circles user vouches for you. circles is unique in that it does not attempt to create a "globally verifiable id"; rather, it creates a graph of trust relationships, where someone's trustworthiness can only be verified from the perspective of your own position in that graph. how does worldcoin work? each worldcoin user installs an app on their phone, which generates a private and public key, much like an ethereum wallet. they then go in-person to visit an "orb". the user stares into the orb's camera, and at the same time shows the orb a qr code generated by their worldcoin app, which contains their public key. the orb scans the user's eyes, and uses complicated hardware scanning and machine-learned classifiers to verify that: the user is a real human the user's iris does not match the iris of any other user that has previously used the system if both scans pass, the orb signs a message approving a specialized hash of the user's iris scan. the hash gets uploaded to a database currently a centralized server, intended to be replaced with a decentralized on-chain system once they are sure the hashing mechanism works. the system does not store full iris scans; it only stores hashes, and these hashes are used to check for uniqueness. from that point forward, the user has a "world id". a world id holder is able to prove that they are a unique human by generating a zk-snark proving that they hold the private key corresponding to a public key in the database, without revealing which key they hold. hence, even if someone re-scans your iris, they will not be able to see any actions that you have taken. what are the major issues with worldcoin's construction? there are four major risks that immediately come to mind: privacy. the registry of iris scans may reveal information. at the very least, if someone else scans your iris, they can check it against the database to determine whether or not you have a world id. potentially, iris scans might reveal more information. accessibility. world ids are not going to be reliably accessible unless there are so many orbs that anyone in the world can easily get to one. centralization. the orb is a hardware device, and we have no way to verify that it was constructed correctly and does not have backdoors. hence, even if the software layer is perfect and fully decentralized, the worldcoin foundation still has the ability to insert a backdoor into the system, letting it create arbitrarily many fake human identities. security. users' phones could be hacked, users could be coerced into scanning their irises while showing a public key that belongs to someone else, and there is the possibility of 3d-printing "fake people" that can pass the iris scan and get world ids. it's important to distinguish between (i) issues specific to choices made by worldcoin, (ii) issues that any biometric proof of personhood will inevitably have, and (iii) issues that any proof of personhood in general will have. for example, signing up to proof of humanity means publishing your face on the internet. joining a brightid verification party doesn't quite do that, but still exposes who you are to a lot of people. and joining circles publicly exposes your social graph. worldcoin is significantly better at preserving privacy than either of those. on the other hand, worldcoin depends on specialized hardware, which opens up the challenge of trusting the orb manufacturers to have constructed the orbs correctly a challenge which has no parallels in proof of humanity, brightid or circles. it's even conceivable that in the future, someone other than worldcoin will create a different specialized-hardware solution that has different tradeoffs. how do biometric proof-of-personhood schemes address privacy issues? the most obvious, and greatest, potential privacy leak that any proof-of-personhood system has is linking each action that a person takes to a real-world identity. this data leak is very large, arguably unacceptably large, but fortunately it is easy to solve with zero knowledge proof technology. instead of directly making a signature with a private key whose corresponding public key is in the database, a user could make a zk-snark proving that they own the private key whose corresponding public key is somewhere in the database, without revealing which specific key they have. this can be done generically with tools like sismo (see here for the proof of humanity-specific implementation), and worldcoin has its own built-in implementation. it's important to give "crypto-native" proof of personhood credit here: they actually care about taking this basic step to provide anonymization, whereas basically all centralized identity solutions do not. a more subtle, but still important, privacy leak is the mere existence of a public registry of biometric scans. in the case of proof of humanity, this is a lot of data: you get a video of each proof of humanity participant, making it very clear to anyone in the world who cares to investigate who all the proof of humanity participants are. in the case of worldcoin, the leak is much more limited: the orb locally computes and publishes only a "hash" of each person's iris scan. this hash is not a regular hash like sha256; rather, it is a specialized algorithm based on machine-learned gabor filters that deals with the inexactness inherent in any biometric scan, and ensures that successive hashes taken of the same person's iris have similar outputs. blue: percent of bits that differ between two scans of the same person's iris. orange: percent of bits that differ between two scans of two different people's irises. these iris hashes leak only a small amount of data. if an adversary can forcibly (or secretly) scan your iris, then they can compute your iris hash themselves, and check it against the database of iris hashes to see whether or not you participated in the system. this ability to check whether or not someone signed up is necessary for the system itself to prevent people from signing up multiple times, but there's always the possibility that it will somehow be abused. additionally, there is the possibility that the iris hashes leak some amount of medical data (sex, ethnicity, perhaps medical conditions), but this leak is far smaller than what could be captured by pretty much any other mass data-gathering system in use today (eg. even street cameras). on the whole, to me the privacy of storing iris hashes seems sufficient. if others disagree with this judgement and decide that they want to design a system with even more privacy, there are two ways to do so: if the iris hashing algorithm can be improved to make the difference between two scans of the same person much lower (eg. reliably under 10% bit flips), then instead of storing full iris hashes, the system can store a smaller number of error correction bits for iris hashes (see: fuzzy extractors). if the difference between two scans is under 10%, then the number of bits that needs to be published would be at least 5x less. if we want to go further, we could store the iris hash database inside a multi-party computation (mpc) system which could only be accessed by orbs (with a rate limit), making the data unaccessible entirely, but at the cost of significant protocol complexity and social complexity in governing the set of mpc participants. this would have the benefit that users would not be able to prove a link between two different world ids that they had at different times even if they wanted to. unfortunately, these techniques are not applicable to proof of humanity, because proof of humanity requires the full video of each participant to be publicly available so that it can be challenged if there are signs that it is fake (including ai-generated fakes), and in such cases investigated in more detail. on the whole, despite the "dystopian vibez" of staring into an orb and letting it scan deeply into your eyeballs, it does seem like specialized hardware systems can do quite a decent job of protecting privacy. however, the flip side of this is that specialized hardware systems introduce much greater centralization concerns. hence, we cypherpunks seem to be stuck in a bind: we have to trade off one deeply-held cypherpunk value against another. what are the accessibility issues in biometric proof-of-personhood systems? specialized hardware introduces accessibility concerns because, well, specialized hardware is not very accessible. somewhere between 51% and 64% of sub-saharan africans now have smartphones, and this seems to be projected to increase to 87% by 2030. but while there are billions of smartphones, there are only a few hundred orbs. even with much higher-scale distributed manufacturing, it would be hard to get to a world where there's an orb within five kilometers of everyone. but to the team's credit, they have been trying! it is also worth noting that many other forms of proof of personhood have accessibility problems that are even worse. it is very difficult to join a social-graph-based proof-of-personhood system unless you already know someone who is in the social graph. this makes it very easy for such systems to remain restricted to a single community in a single country. even centralized identity systems have learned this lesson: india's aadhaar id system is biometric-based, as that was the only way to quickly onboard its massive population while avoiding massive fraud from duplicate and fake accounts (resulting in huge cost savings), though of course the aadhaar system as a whole is far weaker on privacy than anything being proposed on a large scale within the crypto community. the best-performing systems from an accessibility perspective are actually systems like proof of humanity, which you can sign up to using only a smartphone though, as we have seen and as we will see, such systems come with all kinds of other tradeoffs. what are the centralization issues in biometric proof-of-personhood systems? there are three: centralization risks in the system's top-level governance (esp. the system that makes final top-level resolutions if different actors in the system disagree on subjective judgements). centralization risks unique to systems that use specialized hardware. centralization risks if proprietary algorithms are used to determine who is an authentic participant. any proof-of-personhood system must contend with (1), perhaps with the exception of systems where the set of "accepted" ids is completely subjective. if a system uses incentives denominated in outside assets (eg. eth, usdc, dai), then it cannot be fully subjective, and so governance risks become unavoidable. [2] is a much bigger risk for worldcoin than for proof of humanity (or brightid), because worldcoin depends on specialized hardware and other systems do not. [3] is a risk particularly in "logically centralized" systems where there is a single system doing the verification, unless all of the algorithms are open-source and we have an assurance that they are actually running the code that they claim they are. for systems that rely purely on users verifying other users (like proof of humanity), it is not a risk. how does worldcoin address hardware centralization issues? currently, a worldcoin-affiliated entity called tools for humanity is the only organization that is making orbs. however, the orb's source code is mostly public: you can see the hardware specs in this github repository, and other parts of the source code are expected to be published soon. the license is another one of those "shared source but not technically open source until four years from now" licenses similar to the uniswap bsl, except in addition to preventing forking it also prevents what they consider unethical behavior they specifically list mass surveillance and three international civil rights declarations. the team's stated goal is to allow and encourage other organizations to create orbs, and over time transition from orbs being created by tools for humanity to having some kind of dao that approves and manages which organizations can make orbs that are recognized by the system. there are two ways in which this design can fail: it fails to actually decentralize. this could happen because of the common trap of federated protocols: one manufacturer ends up dominating in practice, causing the system to re-centralize. presumably, governance could limit how many valid orbs each manufacturer can produce, but this would need to be managed carefully, and it puts a lot of pressure on governance to be both decentralized and monitor the ecosystem and respond to threats effectively: a much harder task than eg. a fairly static dao that just handles top-level dispute resolution tasks. it turns out that it's not possible to make such a distributed manufacturing mechanism secure. here, there are two risks that i see: fragility against bad orb manufacturers: if even one orb manufacturer is malicious or hacked, it can generate an unlimited number of fake iris scan hashes, and give them world ids. government restriction of orbs: governments that do not want their citizens participating in the worldcoin ecosystem can ban orbs from their country. furthermore, they could even force their citizens to get their irises scanned, allowing the government to get their accounts, and the citizens would have no way to respond. to make the system more robust against bad orb manufacturers, the worldcoin team is proposing to perform regular audits on orbs, verifying that they are built correctly and key hardware components were built according to specs and were not tampered with after the fact. this is a challenging task: it's basically something like the iaea nuclear inspections bureaucracy but for orbs. the hope is that even a very imperfect implementation of an auditing regime could greatly cut down on the number of fake orbs. to limit the harm caused by any bad orb that does slip through, it makes sense to have a second mitigation. world ids registered with different orb manufacturers, and ideally with different orbs, should be distinguishable from each other. it's okay if this information is private and only stored on the world id holder's device; but it does need to be provable on demand. this makes it possible for the ecosystem to respond to (inevitable) attacks by removing individual orb manufacturers, and perhaps even individual orbs, from the whitelist on-demand. if we see the north korea government going around and forcing people to scan their eyeballs, those orbs and any accounts produced by them could be immediately retroactively disabled. security issues in proof of personhood in general in addition to issues specific to worldcoin, there are concerns that affect proof-of-personhood designs in general. the major ones that i can think of are: 3d-printed fake people: one could use ai to generate photographs or even 3d prints of fake people that are convincing enough to get accepted by the orb software. if even one group does this, they can generate an unlimited number of identities. possibility of selling ids: someone can provide someone else's public key instead of their own when registering, giving that person control of their registered id, in exchange for money. this seems to be happening already. in addition to selling, there's also the possibility of renting ids to use for a short time in one application. phone hacking: if a person's phone gets hacked, the hacker can steal the key that controls their world id. government coercion to steal ids: a government could force their citizens to get verified while showing a qr code belonging to the government. in this way, a malicious government could gain access to millions of ids. in a biometric system, this could even be done covertly: governments could use obfuscated orbs to extract world ids from everyone entering their country at the passport control booth. [1] is specific to biometric proof-of-personhood systems. [2] and [3] are common to both biometric and non-biometric designs. [4] is also common to both, though the techniques that are required would be quite different in both cases; in this section i will focus on the issues in the biometric case. these are pretty serious weaknesses. some already have been addressed in existing protocols, others can be addressed with future improvements, and still others seem to be fundamental limitations. how can we deal with fake people? this is significantly less of a risk for worldcoin than it is for proof of humanity-like systems: an in-person scan can examine many features of a person, and is quite hard to fake, compared to merely deep-faking a video. specialized hardware is inherently harder to fool than commodity hardware, which is in turn harder to fool than digital algorithms verifying pictures and videos that are sent remotely. could someone 3d-print something that can fool even specialized hardware eventually? probably. i expect that at some point we will see growing tensions between the goal of keeping the mechanism open and keeping it secure: open-source ai algorithms are inherently more vulnerable to adversarial machine learning. black-box algorithms are more protected, but it's hard to tell that a black-box algorithm was not trained to include backdoors. perhaps zk-ml technologies could give us the best of both worlds. though at some point in the even further future, it is likely that even the best ai algorithms will be fooled by the best 3d-printed fake people. however, from my discussions with both the worldcoin and proof of humanity teams, it seems like at the present moment neither protocol is yet seeing significant deep fake attacks, for the simple reason that hiring real low-wage workers to sign up on your behalf is quite cheap and easy. can we prevent selling ids? in the short term, preventing this kind of outsourcing is difficult, because most people in the world are not even aware of proof-of-personhood protocols, and if you tell them to hold up a qr code and scan their eyes for $30 they will do that. once more people are aware of what proof-of-personhood protocols are, a fairly simple mitigation becomes possible: allowing people who have a registered id to re-register, canceling the previous id. this makes "id selling" much less credible, because someone who sells you their id can just go and re-register, canceling the id that they just sold. however, getting to this point requires the protocol to be very widely known, and orbs to be very widely accessible to make on-demand registration practical. this is one of the reasons why having a ubi coin integrated into a proof-of-personhood system is valuable: a ubi coin provides an easily understandable incentive for people to (i) learn about the protocol and sign up, and (ii) immediately re-register if they register on behalf of someone else. re-registration also prevents phone hacking. can we prevent coercion in biometric proof-of-personhood systems? this depends on what kind of coercion we are talking about. possible forms of coercion include: governments scanning people's eyes (or faces, or...) at border control and other routine government checkpoints, and using this to register (and frequently re-register) their citizens governments banning orbs within the country to prevent people from independently re-registering individuals buying ids and then threatening to harm the seller if they detect that the id has been invalidated due to re-registration (possibly government-run) applications requiring people to "sign in" by signing with their public key directly, letting them see the corresponding biometric scan, and hence the link between the user's current id and any future ids they get from re-registering. a common fear is that this makes it too easy to create "permanent records" that stick with a person for their entire life. all your ubi and voting power are belong to us. image source. especially in the hands of unsophisticated users, it seems quite tough to outright prevent these situations. users could leave their country to (re-)register at an orb in a safer country, but this is a difficult process and high cost. in a truly hostile legal environment, seeking out an independent orb seems too difficult and risky. what is feasible is making this kind of abuse more annoying to implement and detectable. the proof of humanity approach of requiring a person to speak a specific phrase when registering is a good example: it may be enough to prevent hidden scanning, requiring coercion to be much more blatant, and the registration phrase could even include a statement confirming that the respondent knows that they have the right to re-register independently and may get ubi coin or other rewards. if coercion is detected, the devices used to perform coercive registrations en masse could have their access rights revoked. to prevent applications linking people's current and previous ids and attempting to leave "permanent records", the default proof of personhood app could lock the user's key in trusted hardware, preventing any application from using the key directly without the anonymizing zk-snark layer in between. if a government or application developer wants to get around this, they would need to mandate the use of their own custom app. with a combination of these techniques and active vigilance, locking out those regimes that are truly hostile, and keeping honest those regimes that are merely medium-bad (as much of the world is), seems possible. this can be done either by a project like worldcoin or proof of humanity maintaining its own bureaucracy for this task, or by revealing more information about how an id was registered (eg. in worldcoin, which orb it came from), and leaving this classification task to the community. can we prevent renting ids (eg. to sell votes)? renting out your id is not prevented by re-registration. this is okay in some applications: the cost of renting out your right to collect the day's share of ubi coin is going to be just the value of the day's share of ubi coin. but in applications such as voting, easy vote selling is a huge problem. systems like maci can prevent you from credibly selling your vote, by allowing you to later cast another vote that invalidates your previous vote, in such a way that no one can tell whether or not you in fact cast such a vote. however, if the briber controls which key you get at registration time, this does not help. i see two solutions here: run entire applications inside an mpc. this would also cover the re-registration process: when a person registers to the mpc, the mpc assigns them an id that is separate from, and not linkable to, their proof of personhood id, and when a person re-registers, only the mpc would know which account to deactivate. this prevents users from making proofs about their actions, because every important step is done inside an mpc using private information that is only known to the mpc. decentralized registration ceremonies. basically, implement something like this in-person key-registration protocol that requires four randomly selected local participants to work together to register someone. this could ensure that registration is a "trusted" procedure that an attacker cannot snoop in during. social-graph-based systems may actually perform better here, because they can create local decentralized registration processes automatically as a byproduct of how they work. how do biometrics compare with the other leading candidate for proof of personhood, social graph-based verification? aside from biometric approaches, the main other contender for proof of personhood so far has been social-graph-based verification. social-graph-based verification systems all operate on the same principle: if there are a whole bunch of existing verified identities that all attest to the validity of your identity, then you probably are valid and should also get verified status. if only a few real users (accidentally or maliciously) verify fake users, then you can use basic graph-theory techniques to put an upper bound on how many fake users get verified by the system. source: https://www.sciencedirect.com/science/article/abs/pii/s0045790622000611. proponents of social-graph-based verification often describe it as being a better alternative to biometrics for a few reasons: it does not rely on special-purpose hardware, making it much easier to deploy it avoids a permanent arms race between manufacturers trying to create fake people and the orb needing to be updated to reject such fake people it does not require collecting biometric data, making it more privacy-friendly it is potentially more friendly to pseudonymity, because if someone chooses to split their internet life across multiple identities that they keep separate from each other, both of those identities could potentially be verified (but maintaining multiple genuine and separate identities sacrifices network effects and has a high cost, so it's not something that attackers could do easily) biometric approaches give a binary score of "is a human" or "is not a human", which is fragile: people who are accidentally rejected would end up with no ubi at all, and potentially no ability to participate in online life. social-graph-based approaches can give a more nuanced numerical score, which may of course be moderately unfair to some participants but is unlikely to "un-person" someone completely. my perspective on these arguments is that i largely agree with them! these are genuine advantages of social-graph-based approaches and should be taken seriously. however, it's worth also taking into account the weaknesses of social-graph-based approaches: bootstrapping: for a user to join a social-graph-based system, that user must know someone who is already in the graph. this makes large-scale adoption difficult, and risks excluding entire regions of the world that do not get lucky in the initial bootstrapping process. privacy: while social-graph-based approaches avoid collecting biometric data, they often end up leaking info about a person's social relationships, which may lead to even greater risks. of course, zero-knowledge technology can mitigate this (eg. see this proposal by barry whitehat), but the interdependency inherent in a graph and the need to perform mathematical analyses on the graph makes it harder to achieve the same level of data-hiding that you can with biometrics. inequality: each person can only have one biometric id, but a wealthy and socially well-connected person could use their connections to generate many ids. essentially, the same flexibility that might allow a social-graph-based system to give multiple pseudonyms to someone (eg. an activist) that really needs that feature would likely also imply that more powerful and well-connected people can gain more pseudonyms than less powerful and well-connected people. risk of collapse into centralization: most people are too lazy to spend time reporting into an internet app who is a real person and who is not. as a result, there is a risk that the system will come over time to favor "easy" ways to get inducted that depend on centralized authorities, and the "social graph" that the system users will de-facto become the social graph of which countries recognize which people as citizens giving us centralized kyc with needless extra steps. is proof of personhood compatible with pseudonymity in the real world? in principle, proof of personhood is compatible with all kinds of pseudonymity. applications could be designed in such a way that someone with a single proof of personhood id can create up to five profiles within the application, leaving room for pseudonymous accounts. one could even use quadratic formulas: n accounts for a cost of $n². but will they? a pessimist, however, might argue that it is naive to try to create a more privacy-friendly form of id and hope that it will actually get adopted in the right way, because the powers-that-be are not privacy-friendly, and if a powerful actor gets a tool that could be used to get much more information about a person, they will use it that way. in such a world, the argument goes, the only realistic approach is, unfortunately, to throw sand in the gears of any identity solution, and defend a world with full anonymity and digital islands of high-trust communities. i see the reasoning behind this way of thinking, but i worry that such an approach would, even if successful, lead to a world where there's no way for anyone to do anything to counteract wealth concentration and governance centralization, because one person could always pretend to be ten thousand. such points of centralization would, in turn, be easy for the powers-that-be to capture. rather, i would favor a moderate approach, where we vigorously advocate for proof-of-personhood solutions to have strong privacy, potentially if desired even include a "n accounts for $n²" mechanism at protocol layer, and create something that has privacy-friendly values and has a chance of getting accepted by the outside world. so... what do i think? there is no ideal form of proof of personhood. instead, we have at least three different paradigms of approaches that all have their own unique strengths and weaknesses. a comparison chart might look as follows: social-graph-based general-hardware biometric specialized-hardware biometric privacy low fairly low fairly high accessibility / scalability fairly low high medium robustness of decentralization fairly high fairly high fairly low security against "fake people" high (if done well) low medium what we should ideally do is treat these three techniques as complementary, and combine them all. as india's aadhaar has shown at scale, specialized-hardware biometrics have their benefits of being secure at scale. they are very weak at decentralization, though this can be addressed by holding individual orbs accountable. general-purpose biometrics can be adopted very easily today, but their security is rapidly dwindling, and they may only work for another 1-2 years. social-graph-based systems bootstrapped off of a few hundred people who are socially close to the founding team are likely to face constant tradeoffs between completely missing large parts of the world and being vulnerable to attacks within communities they have no visibility into. a social-graph-based system bootstrapped off tens of millions of biometric id holders, however, could actually work. biometric bootstrapping may work better short-term, and social-graph-based techniques may be more robust long-term, and take on a larger share of the responsibility over time as their algorithms improve. a possible hybrid path. all of these teams are in a position to make many mistakes, and there are inevitable tensions between business interests and the needs of the wider community, so it's important to exercise a lot of vigilance. as a community, we can and should push all participants' comfort zones on open-sourcing their tech, demand third-party audits and even third-party-written software, and other checks and balances. we also need more alternatives in each of the three categories. at the same time it's important to recognize the work already done: many of the teams running these systems have shown a willingness to take privacy much more seriously than pretty much any government or major corporate-run identity systems, and this is a success that we should build on. the problem of making a proof-of-personhood system that is effective and reliable, especially in the hands of people distant from the existing crypto community, seems quite challenging. i definitely do not envy the people attempting the task, and it will likely take years to find a formula that works. the concept of proof-of-personhood in principle seems very valuable, and while the various implementations have their risks, not having any proof-of-personhood at all has its risks too: a world with no proof-of-personhood seems more likely to be a world dominated by centralized identity solutions, money, small closed communities, or some combination of all three. i look forward to seeing more progress on all types of proof of personhood, and hopefully seeing the different approaches eventually come together into a coherent whole. dark mode toggle the revenue-evil curve: a different way to think about prioritizing public goods funding 2022 oct 28 see all posts special thanks to karl floersch, hasu and tina zhen for feedback and review. public goods are an incredibly important topic in any large-scale ecosystem, but they are also one that is often surprisingly tricky to define. there is an economist definition of public goods goods that are non-excludable and non-rivalrous, two technical terms that taken together mean that it's difficult to provide them through private property and market-based means. there is a layman's definition of public good: "anything that is good for the public". and there is a democracy enthusiast's definition of public good, which includes connotations of public participation in decision-making. but more importantly, when the abstract category of non-excludable non-rivalrous public goods interacts with the real world, in almost any specific case there are all kinds of subtle edge cases that need to be treated differently. a park is a public good. but what if you add a $5 entrance fee? what if you fund it by auctioning off the right to have a statue of the winner in the park's central square? what if it's maintained by a semi-altruistic billionaire that enjoys the park for personal use, and designs the park around their personal use, but still leaves it open for anyone to visit? this post will attempt to provide a different way of analyzing "hybrid" goods on the spectrum between private and public: the revenue-evil curve. we ask the question: what are the tradeoffs of different ways to monetize a given project, and how much good can be done by adding external subsidies to remove the pressure to monetize? this is far from a universal framework: it assumes a "mixed-economy" setting in a single monolithic "community" with a commercial market combined with subsidies from a central funder. but it can still tell us a lot about how to approach funding public goods in crypto communities, countries and many other real-world contexts today. the traditional framework: excludability and rivalrousness let us start off by understanding how the usual economist lens views which projects are private vs public goods. consider the following examples: alice owns 1000 eth, and wants to sell it on the market. bob runs an airline, and sells tickets for a flight. charlie builds a bridge, and charges a toll to pay for it. david makes and releases a podcast. eve makes and releases a song. fred invents a new and better cryptographic algorithm for making zero knowledge proofs. we put these situations on a chart with two axes: rivalrousness: to what extent does one person enjoying the good reduce another person's ability to enjoy it? excludability: how difficult is it to prevent specific individuals, eg. those who do not pay, from enjoying the good? such a chart might look like this: alice's eth is completely excludable (she has total power to choose who gets her coins), and crypto coins are rivalrous (if one person owns a particular coin, no one else owns that same coin) bob's plane tickets are excludable, but a tiny bit less rivalrous: there's a chance the plane won't be full. charlie's bridge is a bit less excludable than plane tickets, because adding a gate to verify payment of tolls takes extra effort (so charlie can exclude but it's costly, both to him and to users), and its rivalrousness depends on whether the road is congested or not. david's podcast and eve's song are not rivalrous: one person listening to it does not interfere with another person doing the same. they're a little bit excludable, because you can make a paywall but people can circumvent the paywall. and fred's cryptographic algorithm is close to not excludable at all: it needs to be open-source for people to trust it, and if fred tries to patent it, the target user base (open-source-loving crypto users) may well refuse to use the algorithm and even cancel him for it. this is all a good and important analysis. excludability tells us whether or not you can fund the project by charging a toll as a business model, and rivalrousness tells us whether exclusion is a tragic waste or if it's just an unavoidable property of the good in question that if one person gets it another does not. but if we look at some of the examples carefully, especially the digital examples, we start to see that it misses a very important issue: there are many business models available other than exclusion, and those business models have tradeoffs too. consider one particular case: david's podcast versus eve's song. in practice, a huge number of podcasts are released mostly or completely freely, but songs are more often gated with licensing and copyright restrictions. to see why, we need only look at how these podcasts are funded: sponsorships. podcast hosts typically find a few sponsors, and talk about the sponsors briefly at the start or middle of each episode. sponsoring songs is harder: you can't suddenly start talking about how awesome athletic greens* are in the middle of a love song, because come on, it kills the vibe, man! can we get beyond focusing solely on exclusion, and talk about monetization and the harms of different monetization strategies more generally? indeed we can, and this is exactly what the revenue/evil curve is about. the revenue-evil curve, defined the revenue-evil curve of a product is a two-dimensional curve that plots the answer to the following question: how much harm would the product's creator have to inflict on their potential users and the wider community to earn $n of revenue to pay for building the product? the word "evil" here is absolutely not meant to imply that no quantity of evil is acceptable, and that if you can't fund a project without committing evil you should not do it at all. many projects make hard tradeoffs that hurt their customers and community in order to ensure sustainable funding, and often the value of the project existing at all greatly outweighs these harms. but nevertheless, the goal is to highlight that there is a tragic aspect to many monetization schemes, and public goods funding can provide value by giving existing projects a financial cushion that enables them to avoid such sacrifices. here is a rough attempt at plotting the revenue-evil curves of our six examples above: for alice, selling her eth at market price is actually the most compassionate thing she could do. if she sells more cheaply, she will almost certainly create an on-chain gas war, trader hft war, or other similarly value-destructive financial conflict between everyone trying to claim her coins the fastest. selling above market price is not even an option: no one would buy. for bob, the socially-optimal price to sell at is the highest price at which all tickets get sold out. if bob sells below that price, tickets will sell out quickly and some people will not be able to get seats at all even if they really need them (underpricing may have a few countervailing benefits by giving opportunities to poor people, but it is far from the most efficient way to achieve that goal). bob could also sell above market price and potentially earn a higher profit at the cost of selling fewer seats and (from the god's-eye perspective) needlessly excluding people. if charlie's bridge and the road leading to it are uncongested, charging any toll at all imposes a burden and needlessly excludes drivers. if they are congested, low tolls help by reducing congestion and high tolls needlessly exclude people. david's podcast can monetize to some extent without hurting listeners much by adding advertisements from sponsors. if pressure to monetize increases, david would have to adopt more and more intrusive forms of advertising, and truly maxing out on revenue would require paywalling the podcast, a high cost to potential listeners. eve is in the same position as david, but with fewer low-harm options (perhaps selling an nft?). especially in eve's case, paywalling may well require actively participating in the legal apparatus of copyright enforcement and suing infringers, which carries further harms. fred has even fewer monetization options. he could patent it, or potentially do exotic things like auction off the right to choose parameters so that hardware manufacturers that favor particular values would bid on it. all options are high-cost. what we see here is that there are actually many kinds of "evil" on the revenue-evil curve: traditional economic deadweight loss from exclusion: if a product is priced above marginal cost, mutually beneficial transactions that could have taken place do not take place race conditions: congestion, shortages and other costs from products being too cheap. "polluting" the product in ways that make it appealing to a sponsor, but is harmful to a (maybe small, maybe large) degree to listeners. engaging in offensive actions through the legal system, which increases everyone's fear and need to spend money on lawyers, and has all kinds of hard-to-predict secondary chilling effects. this is particularly severe in the case of patenting. sacrificing on principles highly valued by the users, the community and even the people working on the project itself. in many cases, this evil is very context-dependent. patenting is both extremely harmful and ideologically offensive within the crypto space and software more broadly, but this is less true in industries building physical goods: in physical goods industries, most people who realistically can create a derivative work of something patented are going to be large and well-organized enough to negotiate for a license, and capital costs mean that the need for monetization is much higher and hence maintaining purity is harder. to what extent advertisements are harmful depends on the advertiser and the audience: if the podcaster understands the audience very well, ads can even be helpful! whether or not the possibility to "exclude" even exists depends on property rights. but by talking about committing evil for the sake of earning revenue in general terms, we gain the ability to compare these situations against each other. what does the revenue-evil curve tell us about funding prioritization? now, let's get back to the key question of why we care about what is a public good and what is not: funding prioritization. if we have a limited pool of capital that is dedicated to helping a community prosper, which things should we direct funding to? the revenue-evil curve graphic gives us a simple starting point for an answer: direct funds toward those projects where the slope of the revenue-evil curve is the steepest. we should focus on projects where each $1 of subsidies, by reducing the pressure to monetize, most greatly reduces the evil that is unfortunately required to make the project possible. this gives us roughly this ranking: top of the line are "pure" public goods, because often there aren't any ways to monetize them at all, or if there are, the economic or moral costs of trying to monetize are extremely high. second priority is "naturally" public but monetizable goods that can be funded through commercial channels by tweaking them a bit, like songs or sponsorships to a podcast. third priority is non-commodity-like private goods where social welfare is already optimized by charging a fee, but where profit margins are high or more generally there are opportunities to "pollute" the product to increase revenue, eg. by keeping accompanying software closed-source or refusing to use standards, and subsidies could be used to push such projects to make more pro-social choices on the margin. notice that the excludability and rivalrousness framework usually outputs similar answers: focus on non-excludable and non-rivalrous goods first, excludable goods but non-rivalrous second, and excludable and partially rivalrous goods last and excludable and rivalrous goods never (if you have capital left over, it's better to just give it out as a ubi). there is a rough approximate mapping between revenue/evil curves and excludability and rivalrousness: higher excludability means lower slope of the revenue/evil curve, and rivalrousness tells us whether the bottom of the revenue/evil curve is zero or nonzero. but the revenue/evil curve is a much more general tool, which allows us to talk about tradeoffs of monetization strategies that go far beyond exclusion. one practical example of how this framework can be used to analyze decision-making is wikimedia donations. i personally have never donated to wikimedia, because i've always thought that they could and should fund themselves without relying on limited public-goods-funding capital by just adding a few advertisements, and this would be only a small cost to their user experience and neutrality. wikipedia admins, however, disagree; they even have a wiki page listing their arguments why they disagree. we can understand this disagreement as a dispute over revenue-evil curves: i think wikimedia's revenue-evil curve has a low slope ("ads are not that bad"), and therefore they are low priority for my charity dollars; some other people think their revenue-evil curve has a high slope, and therefore they are high priority for their charity dollars. revenue-evil curves are an intellectual tool, not a good direct mechanism one important conclusion that it is important not to take from this idea is that we should try to use revenue-evil curves directly as a way of prioritizing individual projects. there are severe constraints on our ability to do this because of limits to monitoring. if this framework is widely used, projects would have an incentive to misrepresent their revenue-evil curves. anyone charging a toll would have an incentive to come up with clever arguments to try to show that the world would be much better if the toll could be 20% lower, but because they're desperately under-budget, they just can't lower the toll without subsidies. projects would have an incentive to be more evil in the short term, to attract subsidies that help them become less evil. for these reasons, it is probably best to use the framework not as a way to allocate decisions directly, but to identify general principles for what kinds of projects to prioritize funding for. for example, the framework can be a valid way to determine how to prioritize whole industries or whole categories of goods. it can help you answer questions like: if a company is producing a public good, or is making pro-social but financially costly choices in the design of a not-quite-public good, do they deserve subsidies for that? but even here, it's better to treat revenue-evil curves as a mental tool, rather than attempting to precisely measure them and use them to make individual decisions. conclusions excludability and rivalrousness are important dimensions of a good, that have really important consequences for its ability to monetize itself, and for answering the question of how much harm can be averted by funding it out of some public pot. but especially once more complex projects enter the fray, these two dimensions quickly start to become insufficient for determining how to prioritize funding. most things are not pure public goods: they are some hybrid in the middle, and there are many dimensions on which they could become more or less public that do not easily map to "exclusion". looking at the revenue-evil curve of a project gives us another way of measuring the statistic that really matters: how much harm can be averted by relieving a project of one dollar of monetization pressure? sometimes, the gains from relieving monetization pressure are decisive: there just is no way to fund certain kinds of things through commercial channels, until you can find one single user that benefits from them enough to fund them unilaterally. other times, commercial funding options exist, but have harmful side effects. sometimes these effects are smaller, sometimes they are greater. sometimes a small piece of an individual project has a clear tradeoff between pro-social choices and increasing monetization. and, still other times, projects just fund themselves, and there is no need to subsidize them or at least, uncertainties and hidden information make it too hard to create a subsidy schedule that does more good than harm. it's always better to prioritize funding in order of greatest gains to smallest; and how far you can go depends on how much funding you have. * i did not accept sponsorship money from athletic greens. but the podcaster lex fridman did. and no, i did not accept sponsorship money from lex fridman either. but maybe someone else did. whatevs man, as long as we can keep getting podcasts funded so they can be free-to-listen without annoying people too much, it's all good, you know? dark mode toggle gitcoin grants round 7 retrospective 2020 oct 18 see all posts round 7 of gitcoin grants has successfully completed! this round has seen an unprecedented growth in interest and contributions, with $274,830 in contributions and $450,000 in matched funds distributed across 857 projects. the category structure was once again changed; this time was had a split between "dapp tech", "infrastructure tech" and "community". here are the results: defi joins the matching! in this round, we were able to have much higher matching values than before. this was because the usual matchings, provided by the ethereum foundation and a few other actors, were supplemented for the first time by a high level of participation from various defi projects: the matchers were: chainlink, a smart contract oracle project optimism, a layer-2 optimistic rollup the ethereum foundation balancer, a decentralized exchange synthetix, a synthetic assets platform yearn, a collateralized-lending platform three arrows capital, an investment fund defiance capital, another investment fund future fund, which is totally not an investment fund! (/s) $meme, a memecoin yam, a defi project some individual contributors: ferretpatrol, bantg, mariano conti, robert leshner, eric conner, 10b576da0 the projects together contributed a large amount of matching funding, some of which was used this round and some of which is reserved as a "rainy day fund" for future rounds in case future matchers are less forthcoming. this is a significant milestone for the ecosystem because it shows that gitcoin grants is expanding beyond reliance on a very small number of funders, and is moving toward something more sustainable. but it is worth exploring, what exactly is driving these matchers to contribute, and is it sustainable? there are a few possibile motivations that are likely all in play to various extents: people are naturally altruistic to some extent, and this round defi projects got unexpectedly wealthy for the first time due to a rapid rise in interest and token prices, and so donating some of that windfall felt like a natural "good thing to do" many in the community are critical of defi projects by default, viewing them as unproductive casinos that create a negative image of what ethereum is supposed to be about. contributing to public goods is an easy way for a defi project to show that they want to be a positive contributor to the ecosystem and make it better even in the absence of such negative perceptions, defi is a competitive market that is heavily dependent on community support and network effects, and so it's very valuable to a project to win friends in the ecosystem the largest defi projects capture enough of the benefit from these public goods that it's in their own interest to contribute there's a high degree of common-ownership between defi projects (holders of one token also hold other tokens and hold eth), and so even if it's not strictly in a project's interest to donate a large amount, token holders of that project push the project to contribute because they as holders benefit from the gains to both that project but also to the other projects whose tokens they hold. the remaining question is, of course: how sustainable will these incentives be? are the altruistic and public-relations incentives only large enough for a one-time burst of donations of this size, or could it become more sustainable? could we reliably expect to see, say, $2-3 million per year spent on quadratic funding matching from here on? if so, it would be excellent news for public goods funding diversification and democratization in the ethereum ecosystem. where did the troublemakers go? one curious result from the previous round and this round is that the "controversial" community grant recipients from previous rounds seem to have dropped in prominence on their own. in theory, we should have seen them continue to get support from their supporters with their detractors being able to do nothing about it. in practice, though, the top media recipients this round appear to be relatively uncontroversial and universally beloved mainstays of the ethereum ecosystem. even the zero knowledge podcast, an excellent podcast but one aimed for a relatively smaller and more highly technical audience, has received a large contribution this round. what happened? why did the distribution of media recipients improve in quality all on its own? is the mechanism perhaps more self-correcting than we had thought? overpayment this round is the first round where top recipients on all sides received quite a large amount. on the infrastructure side, the white hat hacking project (basically a fund to donate to samczsun) received a total of $39,258, and the bankless podcast got $47,620. we could ask the question: are the top recipients getting too much funding? to be clear, i do think that it's very improper to try to create a moral norm that public goods contributors should only be earning salaries up to a certain level and should not be able to earn much more than that. people launching coins earn huge windfalls all the time; it is completely natural and fair for public goods contributors to also get that possibility (and furthermore, the numbers from this round translate to about ~$200,000 per year, which is not even that high). however, one can ask a more limited and pragmatic question: given the current reward structure, is putting an extra $1 into the hands of a top contributor less valuable than putting $1 into the hands of one of the other very valuable projects that's still underfunded? turbogeth, nethermind, radicalxchange and many other projects could still do quite a lot with a marginal dollar. for the first time, the matching amounts are high enough that this is actually a significant issue. especially if matching amounts increase even further, is the ecosystem going to be able to correctly allocate funds and avoid overfunding projects? alternatively, if it fails to avoid over-concentrating funds, is that all that bad? perhaps the possibility of becoming the center of attention for one round and earning a $500,000 windfall will be part of the incentive that motivates independent public goods contributors! we don't know; but these are the yet-unknown facts that running the experiment at its new increased scale is for the first time going to reveal. let's talk about categories... the concept of categories as it is currently implemented in gitcoin grants is a somewhat strange one. each category has a fixed total matching amount that is split between projects within that category. what this mechanism basically says is that the community can be trusted to choose between projects within a category, but we need a separate technocratic judgement to judge how the funds are split between the different categories in the first place. but it gets more paradoxical from here. in round 7, a "collections" feature was introduced halfway through the round: if you click "add to cart" on a collection, you immediately add everything in the collection to your cart. this is strange because this mechanism seems to send the exact opposite message: users that don't understand the details well can choose to allocate funds to entire categories, but (unless they manually edit the amounts) they should not be making many active decisions within each category. which is it? do we trust the radical quadratic fancy democracy to allocate within categories but not between them, do we trust it to allocate between categories but nudge people away from making fine-grained decisions within them, or something else entirely? i recommend that for round 8 we think harder about the philosophical challenges here and come up with a more principled approach. one option would be to have one matching pool and have all the categories just be a voluntary ui layer. another would be to experiment with even more "affirmative action" to bootstrap particular categories: for example, we could split the community matching into a $25,000 matching pool for each major world region (eg. north america + oceania, latin america, europe, africa, middle east, india, east + southeast asia) to try to give projects in more neglected areas a leg up. there are many possibilities here! one hybrid route is that the "focused" pools could themselves be quadratic funded in the previous round! identity verification as collusion, fake accounts and other attacks on gitcoin grants have been recently increasing, round 7 added an additional verification option with the decentralized social-graph-based brightid, and single-handedly boosted the project's userbase by a factor of ten: this is good, because along with helping brightid's growth, it also subjects the project to a trial-by-fire: there's now a large incentive to try to create a large number of fake accounts on it! brightid is going to face a tough challenge making it reasonably easy for regular users to join but at the same time resist attacks from fake and duplicate accounts. i look forward to seeing them try to meet the challenge! zk rollups for scalability finally, round 7 was the first round where gitcoin grants experimented with using the zksync zk rollup to decrease fees for payments: the main thing to report here is simply that the zk rollup successfully did decrease fees! the user experience worked well. many optimistic and zk rollup projects are now looking at collaborating with wallets on direct integrations, which should increase the usability and security of such techniques further. conclusions round 7 has been a pivotal round for gitcoin grants. the matching funding has become much more sustainable. the levels of funding are now large enough to successfully fund quadratic freelancers to the point where a project getting "too much funding" is a conceivable thing to worry about! identity verification is taking steps forward. payments have become much more efficient with the introduction of the zksync zk rollup. i look forward to seeing the grants continue for many more rounds in the future. bazaar services | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search bazaar services posted by gavin wood on april 5, 2015 research & development i'm a great believer in judging the present through the knowledge of the past. in the past 25 or so years one of the biggest trends in business and technology was what might loosely be termed as the open source revolution. the notion of it being good business to share your source code might not yet be cemented into the eyes of many a traditionalist, however the idea of leveraging existing open source software is. we are swiftly getting to the point where nearly all important, non-niche software is, one way or another, open. in all android phones, all mac computers, practically all mainstream web technologies: servers, databases, browsers, compilers; all of the foundations are open. this is in stark contrast to when i was writing my degree dissertation "open source software in the business environment", shortly after esr wrote "the cathedral and the bazaar", when microsoft and its gigantic closed source codebase was the undisputed leader and there were but one or two serious examples of open source software being used commercially. so why? what changed? did people suddenly realise that, as raymond put it, the "bazaar" model was the right way to go? no. the introduction of ideas alone rarely make the difference and in any case, the notion of a distributed workforce, through individual interest working on a cohesive whole is not exactly ground breaking. in fact, software development, as a process, was always perfect for decentralisation the one thing is lacked was a ubiquitous communications infrastructure for the developers some way for them to share code seamlessly and work together easily. it is no surprise that the rise of the internet with cvs, irc, usenet and mailing lists coincided with that of open source software. so was much else provided by the previous "cathedral" model? actually, yes. it facilitated a bunch of business stuff which we might loosely consider "value plumbing"; firstly, it incentivised the practitioners it paid developers to spend their time and energy on a project. secondly, it provided all necessary support assets to allow the development to take place (hardware, software, tools, educational material &c.). thirdly, it acted as a sink for funds it collected payments from those who benefited from the work being done. in a nutshell, it looked after cash flow, scooping a tidy profit in return for enabling and incentivising solution production. assumed at first to be important, it turned out less so; turns out people will often work on software just for the fun of it. nonetheless, we cannot deny that this "value plumbing" still pays an important role in human activity and service provision. so what does this tell us about the future? business, and in particular the service industry, has, until now, followed very much a "cathedral" approach to the service commission, provision and management that would fall under this "value plumbing". we might recognise it by tight coordination, enforced consistency, explicit top-down management, centralisation and rigidity. the fact we have singular legal entities with authority and responsibility over large swathes of production output is a very clear signal of this. ebay, as an enabling platform, was a trailblazer in truly global decentralisation; it functioned as a key enabler for small businesses and cottage industries throughout the developed world (not to mention a lucrative source of funds for some of the less scrupulous operators in internet-enabled parts of the developing world). with the web 2.0 platform, and mobile (to which is it heavily entangled), we see a new class of decentralising applications. the so called "sharing economy" is beginning to form with uber, airbnb and taskrabbit as notable examples. much like ebay, these operators reduce the relevance of a whole class of "structural intermediaries", and replace their "value-plumbing" with one large, technologically-adept match-maker. the high-level deconstruction that they entail typically comes with some accidental degree of openness (uber's "safe-driver fee", airbnb's "cleaning charge"; it typical to know more about your matched service provider). so what do taxi firms, hostelries and un-/semi-skilled labour outfits have in common that made them key examples to the "decentralised services"? from where do their profits come that allows them to be reduced to a scalable automaton so easily? they manage their reputation (through basic word of mouth, marketing and advertising), they manage their workforce (through finance, recruiting and agreements) the manage their market (through adapting to changing levels of supply and demand) and they manage their risks (vetting, indemnity, insurance and bonds). though they cannot seriously claim to have created truly new or open markets, they are coming close. in the world of open source software they're a sort of shareware. not quite commercial, but not really free either. there are still singular entities, match-makers, behind the decentralised veneer, as is evident when you go to statist and technosocial-lagard germany and see that the only kind of uber you can request is a regular government-sanctioned taxi. so though these are not quite there yet, these are the beginnings of a social shift in expectations; as consumers we expect greater transparency in the operations of our provider (from knowing the name of our driver to the precise origin of the rubber in our trainers) and greater freedom over the selection of our service; as individuals we expect greater ability to sell our skills, time, possessions or potential; as businesses we expect reduced barriers to entry in whatever markets we wish to compete. just like in open-source software, it won't be long before legions of sufficiently-good amateurs (or professionals seeking to go it alone) will be competing, in a bazaar fashion, on an equal or greater footing to the cathedrals of industry. and the idea of "bazaar services" is the eventual conclusion to this social shift. as open source software is practically zero-barrier to entry and fluid in terms of leadership and authority, so we will find the world of service provision following. the worries are the same as they were 20 years ago. the answers are similar. software writing was just the first thing to go fundamentally decentral, and only because of the natural tech-savviness of its people and its nature of being entirely information-based. with ethereum, crypto-law, web 3.0 and the ilk, all aspects of services will follow the same route. the idea of a rigid organisation or corporation will evaporate and left will be the true essence of human interaction patterns, policed only by openness and information-theoretic mathematics. whereas once the "interaction-pattern-manager", "value-plumber"---or "corporation" for short---would be subject to laws on the emergent behaviour it was enabling, strict legality of the emergent behaviour will become increasingly less relevant as it becomes drastically pluralistic and unpoliceable with no entity, legal or otherwise, coordinating it or profiting from it. we will begin to see a world without middlemen, intermediaries, trusted authorities, where services are not only delivered, but also advertised, found, matched and insured, directly from provider to consumer. interaction patterns arise and continue not through the clumsy, inefficient legal system and slow and rigid corporate rules, but rather through the inherently adaptable emergent effects of flexible, agile and direct economic incentives. this is where we're heading and if it delivers as well as open source software has, it surely can't come fast enough. can we make profits from this new social model? my opinion is a resounding yes; profits will come, as always, from servicing (perceived) human needs or providing efficiency gains to those incentivised to recognise and deploy them. however the sorts of models that are lucrative are not yet obvious. don't expect the profit-making entity to look like any now, or you'll be stuck like those searching for the next microsoft back in 2000 and placing their money in vmlinux and redhat. what we think are great differentiators now will turn out to be commodity in 20 years, much like operating systems and browsers back in '95. to understand where these differentiators may lie first requires understanding what will be made commodity. some food for thought: what if "commodity" turns out to be a digital marketplace, fully vendor neutral, open and trustless? ubiquitous "value-plumbing", business logic constantly evolving yet maintaining readiness that everybody, whoever they are, be able to take part in... bazaar services. 5/4/2015 addendum: having re-visited catb, i must apologise for my abuse of raymond's analogies; the original work was more about the distinction between traditional working practices (mostly commercial software, but it did include some oss such as gnu) and decentralised working practices (what we now tend to think of as open source software development), rather than strictly proprietary/commercial vs floss. still in circumspect terms, the notion of decentralisation within software development is prevalent in and correlated to open source development. interestingly, even aside from oss, some of the agile methodology (i'm thinking scrum) could be argued to be joining this general trend towards the self-organisation, decentralisation and no-authority-operation of the bazaar. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle soulbound 2022 jan 26 see all posts one feature of world of warcraft that is second nature to its players, but goes mostly undiscussed outside of gaming circles, is the concept of soulbound items. a soulbound item, once picked up, cannot be transferred or sold to another player. most very powerful items in the game are soulbound, and typically require completing a complicated quest or killing a very powerful monster, usually with the help of anywhere from four to thirty nine other players. hence, in order to get your character anywhere close to having the best weapons and armor, you have no choice but to participate in killing some of these extremely difficult monsters yourself. the purpose of the mechanic is fairly clear: it keeps the game challenging and interesting, by making sure that to get the best items you have to actually go and do the hard thing and figure out how to kill the dragon. you can't just go kill boars ten hours a day for a year, get thousands of gold, and buy the epic magic armor from other players who killed the dragon for you. of course, the system is very imperfect: you could just pay a team of professionals to kill the dragon with you and let you collect the loot, or even outright buy a character on a secondary market, and do this all with out-of-game us dollars so you don't even have to kill boars. but even still, it makes for a much better game than every item always having a price. what if nfts could be soulbound? nfts in their current form have many of the same properties as rare and epic items in a massively multiplayer online game. they have social signaling value: people who have them can show them off, and there's more and more tools precisely to help users do that. very recently, twitter started rolling out an integration that allows users to show off their nfts on their picture profile. but what exactly are these nfts signaling? certainly, one part of the answer is some kind of skill in acquiring nfts and knowing which nfts to acquire. but because nfts are tradeable items, another big part of the answer inevitably becomes that nfts are about signaling wealth. cryptopunks are now regularly being sold for many millions of dollars, and they are not even the most expensive nfts out there. image source here. if someone shows you that they have an nft that is obtainable by doing x, you can't tell whether they did x themselves or whether they just paid someone else to do x. some of the time this is not a problem: for an nft supporting a charity, someone buying it off the secondary market is sacrificing their own funds for the cause and they are helping the charity by contributing to others' incentive to buy the nft, and so there is no reason to discriminate against them. and indeed, a lot of good can come from charity nfts alone. but what if we want to create nfts that are not just about who has the most money, and that actually try to signal something else? perhaps the best example of a project trying to do this is poap, the "proof of attendance protocol". poap is a standard by which projects can send nfts that represent the idea that the recipient personally participated in some event. part of my own poap collection, much of which came from the events that i attended over the years. poap is an excellent example of an nft that works better if it could be soulbound. if someone is looking at your poap, they are not interested in whether or not you paid someone who attended some event. they are interested in whether or not you personally attended that event. proposals to put certificates (eg. driver's licenses, university degrees, proof of age) on-chain face a similar problem: they would be much less valuable if someone who doesn't meet the condition themselves could just go buy one from someone who does. while transferable nfts have their place and can be really valuable on their own for supporting artists and charities, there is also a large and underexplored design space of what non-transferable nfts could become. what if governance rights could be soulbound? this is a topic i have written about ad nauseam (see [1] [2] [3] [4] [5]), but it continues to be worth repeating: there are very bad things that can easily happen to governance mechanisms if governance power is easily transferable. this is true for two primary types of reasons: if the goal is for governance power to be widely distributed, then transferability is counterproductive as concentrated interests are more likely to buy the governance rights up from everyone else. if the goal is for governance power to go to the competent, then transferability is counterproductive because nothing stops the governance rights from being bought up by the determined but incompetent. if you take the proverb that "those who most want to rule people are those least suited to do it" seriously, then you should be suspicious of transferability, precisely because transferability makes governance power flow away from the meek who are most likely to provide valuable input to governance and toward the power-hungry who are most likely to cause problems. so what if we try to make governance rights non-transferable? what if we try to make a citydao where more voting power goes to the people who actually live in the city, or at least is reliably democratic and avoids undue influence by whales hoarding a large number of citizen nfts? what if dao governance of blockchain protocols could somehow make governance power conditional on participation? once again, a large and fruitful design space opens up that today is difficult to access. implementing non-transferability in practice poap has made the technical decision to not block transferability of the poaps themselves. there are good reasons for this: users might have a good reason to want to migrate all their assets from one wallet to another (eg. for security), and the security of non-transferability implemented "naively" is not very strong anyway because users could just create a wrapper account that holds the nft and then sell the ownership of that. and indeed, there have been quite a few cases where poaps have frequently been bought and sold when an economic rationale was there to do so. adidas recently released a poap for free to their fans that could give users priority access at a merchandise sale. what happened? well, of course, many of the poaps were quickly transferred to the highest bidder. more transfers than items. and not the only time. to solve this problem, the poap team is suggesting that developers who care about non-transferability implement checks on their own: they could check on-chain if the current owner is the same address as the original owner, and they could add more sophisticated checks over time if deemed necessary. this is, for now, a more future-proof approach. perhaps the one nft that is the most robustly non-transferable today is the proof-of-humanity attestation. theoretically, anyone can create a proof-of-humanity profile with a smart contract account that has transferable ownership, and then sell that account. but the proof-of-humanity protocol has a revocation feature that allows the original owner to make a video asking for a profile to be removed, and a kleros court decides whether or not the video was from the same person as the original creator. once the profile is successfully removed, they can re-apply to make a new profile. hence, if you buy someone else's proof-of-humanity profile, your possession can be very quickly taken away from you, making transfers of ownership non-viable. proof-of-humanity profiles are de-facto soulbound, and infrastructure built on top of them could allow for on-chain items in general to be soulbound to particular humans. can we limit transferability without going all the way and basing everything on proof of humanity? it becomes harder, but there are medium-strength approaches that are probably good enough for some use cases. making an nft bound to an ens name is one simple option, if we assume that users care enough about their ens names that they are not willing to transfer them. for now, what we're likely to see is a spectrum of approaches to limit transferability, with different projects choosing different tradeoffs between security and convenience. non-transferability and privacy cryptographically strong privacy for transferable assets is fairly easy to understand: you take your coins, put them into tornado.cash or a similar platform, and withdraw them into a fresh account. but how can we add privacy for soulbound items where you cannot just move them into a fresh account or even a smart contract? if proof of humanity starts getting more adoption, privacy becomes even more important, as the alternative is all of our activity being mapped on-chain directly to a human face. fortunately, a few fairly simple technical options are possible: store the item at an address which is the hash of (i) an index, (ii) the recipient address and (iii) a secret belonging to the recipient. you could reveal your secret to an interface that would then scan for all possible items that belong to your, but no one without your secret could see which items are yours. publish a hash of a bunch of items, and give each recipient their merkle branch. if a smart contract needs to check if you have an item of some type, you can provide a zk-snark. transfers could be done on-chain; the simplest technique may just be a transaction that calls a factory contract to make the old item invalid and the new item valid, using a zk-snark to prove that the operation is valid. privacy is an important part of making this kind of ecosystem work well. in some cases, the underlying thing that the item is representing is already public, and so there is no point in trying to add privacy. but in many other cases, users would not want to reveal everything that they have. if, one day in the future, being vaccinated becomes a poap, one of the worst things we could do would be to create a system where the poap is automatically advertised for everyone to see and everyone has no choice but to let their medical decision be influenced by what would look cool in their particular social circle. privacy being a core part of the design can avoid these bad outcomes and increase the chance that we create something great. from here to there a common criticism of the "web3" space as it exists today is how money-oriented everything is. people celebrate the ownership, and outright waste, of large amounts of wealth, and this limits the appeal and the long-term sustainability of the culture that emerges around these items. there are of course important benefits that even financialized nfts can provide, such as funding artists and charities that would otherwise go unrecognized. however, there are limits to that approach, and a lot of underexplored opportunity in trying to go beyond financialization. making more items in the crypto space "soulbound" can be one path toward an alternative, where nfts can represent much more of who you are and not just what you can afford. however, there are technical challenges to doing this, and an uneasy "interface" between the desire to limit or prevent transfers and a blockchain ecosystem where so far all of the standards are designed around maximum transferability. attaching items to "identity objects" that users are either unable (as with proof-of-humanity profiles) or unwilling (as with ens names) to trade away seems like the most promising path, but challenges remain in making this easy-to-use, private and secure. we need more effort on thinking through and solving these challenges. if we can, this opens a much wider door to blockchains being at the center of ecosystems that are collaborative and fun, and not just about money. the ethereum launch process | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the ethereum launch process posted by vinay gupta on march 3, 2015 research & development i’m vinay gupta, the newly minted release coordinator for ethereum. i’ve been working with the comms team on strategy, and have now come aboard to help smooth the release process (some of the content in this blog is out of date, please see this link for the most up to date information on ethereum). i’ll be about 50/50 on comms and on release coordination. a lot of that is going to be about keeping you updated on progress: new features, new documentation, and hopefully writing about great new services you can use, so it’s in the hinterland between comms and project management. in theory, once i’m up to speed, i should be providing you with the answers to the question: “what’s going on?” but give me some time, because getting up to speed on all of this is nontrivial. we have a very large development team working with very advanced and often quite complex new technology, and keeping everybody up to date on that simultaneously is going to be tricky. to do that well, i have to actually understand what’s going on at quite a technical level first. i have a lot to wrap my head around. i was a 3d graphics programmer through the 1990s, and have a reasonably strong grounding in financial cryptography (i was, and i am not ashamed to admit it, a cypherpunk in those days). but we have a 25-30 person team working in parallel on several different aspects of ethereum, so... patience please while i master the current state of play, so that i can communicate about what’s changing as we move forwards. it’s a lot of context to acquire, as i’m sure you all know if there’s an occasional gaffe as i get oriented, forgive me! i’ve just come back from switzerland, where i got to meet a lot of the team, my “orientation week” being three days during the release planning meetings. gav writes in some detail about that week here, so rather than repeat gav, read his post, and i’ll press on to tell you what was on that release white board. there is good news, there is bad news, but above all, there is a release schedule. there will be another blog post with much more detail about the release schedule for the first live ethereum network shortly likely by the end of this week, as the developer meeting that gav mentions in his post winds up and the conclusions are communicated. that’s the post which will give you timelines you can start firing up your mining rigs to, feature lists, and so on. until then, let me lay out roughly what the four major steps in the release process will look like and we can get into detail soon. let’s lay out where we are first: ethereum is a sprawling project with many teams in many countries implementing the same protocol in several different language versions so it can be integrated into the widest possible range of other systems/ecologies, and to provide long term resilience and future-proofing. in addition to that broad effort, there are several specific applications/toolchains to help people view, build and interact with ethereum: mist, mix, alethzero and so on. starting quite soon, and over the next few months, a series of these tools will be stood up as late alpha, beta, ready for general use and shipped. because the network is valuable, and the network is only as secure as the software we provide, this is going to be a security-led not schedule-led process. you want it done right, we want it done right, and this is one of the most revolutionary software projects ever shipped.  while you’re waiting for the all singing, all dancing cern httpd + ncsa mosaic combo, the “we have just launched the future of the internet” breakthrough system, we will be actually be releasing the code and the tools in layers. we are standing up the infrastructure for a whole new web a piece at a time: server first, plus tool chain, and then the full user experience rich client. this makes sense: a client needs something to connect to, so the server infrastructure has to come first. an internet based on this metacomputer model is going to be a very different place, and getting a good interface to that is going to present a whole new set of challenges. there’s no way to simply put all the pieces together and hope it clips into place like forming an arch by throwing bricks in the air: we need scaffolding, and precise fit. we get that by concentrating on the underlying technical aspects for a while, including mining, the underlying network and so on, and then as that is widely deployed, stable and trusted, we will be moving up the stack towards the graphical user interface via mist in the next few months. none of these pieces stand alone, either: the network needs miners and exchanges, and it takes people time to get organized to do that work properly. the mist client needs applications, or it’s a bare browser with nothing to connect to, and it takes people time to write those applications. each change, each step forwards, involves a lot of conversations and support as we get people set up with the new software and help them get their projects off the ground: the whole thing together is an ecology. each piece needs its own time, its own attention. we have to do this in phases for all of these reasons, and more.  it took bitcoin, a much less complex project, several years to cover that terrain: we have a larger team, but a more complex project. on the other hand, if you’re following the github repositories, you can see how much progress is being made, week by week, day by day, so... verify for yourself where we are. so, now we’ve all got on the same page on real world software engineering, let’s actually look at phases of this release process! release step one: frontier frontier takes a model familiar to bitcoiners, and stands it up for our initial release. frontier is the ethereum network in its barest form: an interface to mine ether, and a way to upload and execute contracts. the main use of frontier on the launch trajectory is to get mining operations and ether exchanges running, so the community can get their mining rigs started, and to start to establish a “live” environment where people can test dapps and acquire ether to upload their own software into ethereum. this is “no user interface to speak of” command line country, and you will be expected to be quite expert in the whole ethereum world model, as well as to have substantial mastery of the tools at your disposal. however, this is not a test net: this is a frontier release. if you are equipped, come along! do not die of dysentery on the way. frontier showcases three areas of real utility: you can mine real ether, at 10% of the normal ether issuance rate, 0.59 ether per block reward, which can be spent to run programs or exchange for other things, as normal this real ether (this was not the case at launch frontier block reward is 5 ether per block, and will remain that amount until casper). you can exchange ether for bitcoin, or with other users, if you need ether to run code etc. if you already bought ether during the crowd sale, and you are fully conversant with the frontier environment, you can use it on the frontier network. we do not recommend this, but have a very substantial security-and-recovery process in place to make it safer see below  we will migrate from frontier to homestead once frontier is fully stable in the eyes of the core devs and the auditors: when we are ready to move to homestead, the release after frontier, the frontier network will be shut down; ether values in wallets will be transferred, but state in contracts is will likely be erased (more information to follow on this in later blog posts) switchover to  the new network will be enforced by “thebomb” this is very early release software: feature complete within these boundaries, but with a substantial risk of unexpected behaviours unseen in either the test net or the security review. and it’s not just us that will be putting new code into production: contracts, exchanges, miners, everybody else in the ecosystem will be shipping new services. any one of those components getting seriously screwed up could impact a lot of users, and we want to shake bugs out of the ecosystem as a whole, not simply our own infrastructure: we are all in this together. however, to help you safeguard your ether, we have the following mechanisms planned (more details from the developers will follow soon as the security model is finalised): if you do not perform any transactions, we guarantee 100% your ether will not be touched and will be waiting for you once we move beyond frontier if you perform transactions, we guarantee 100% that any ether you did not spend will will be available to you once we move beyond frontier not be touched ether you spend will not fall through cracks into other people’s pockets or vanish without a trace: in the unlikely event that this happens, you have 24 hours to inform us, and we will freeze the network, return to the last good state, and start again with the bug patched yes, this implies a real risk of network instability: everything possible has been done to prevent this, but this is a brand new aeroplane take your parachute! we will periodically checkpoint the network to show that neither user report nor automated testing has reported any problems. we expect the checkpoints will be around once daily, with a mean of around 12 hours of latency exchanges etc. will be strongly encouraged to wait for checkpoints to be validated before sending out payments in fiat or bitcoin. ethereum will provide explicit support to aid exchanges in determining what ether transactions have fully cleared over the course of the next few weeks several pieces of software have to be integrated to maintain this basket of security features so we can allow genesis block ether on to this platform without unacceptable risks. building that infrastructure is a new process, and while it looks like a safe, sane and conservative schedule, there is always a chance of a delay as the unknown unknown is discovered either by us, the bug bounty hunters or by the security auditors. there will be a post shortly which goes through this release plan in real technical detail, and i’ll have a lot of direct input from the devs on that post, so for now take this with a pinch of salt and we will have hard details and expected dates as soon as possible.  release step two: homestead homestead is where we move after frontier. we expect the following three major changes. ether mining will be at 100% rather than 10% of the usual reward rate (frontier/homestead block reward will remain 5 ether) checkpointing and manual network halts should never be necessary, although it is likely that checkpointing will continue if there is a general demand for it we will remove the severe risk warning from putting your ether on the network, although we will not consider the software to be out of beta until metropolis still command line, so much the same feature set as frontier, but this one we tell you is ready to go, within the relevant parameters. how long will there be between frontier and homestead? depends entirely on how frontier performs: best case is not less than a month. we will have a pretty good idea of whether things are going smoothly or not from network review, so we will keep you in the loop through this process. release step three: metropolis metropolis is when we finally officially release a relatively full-featured user interface for non-technical users of ethereum, and throw the doors open: mist launches, and we expect this launch to include a dapp store and several anchor tenant projects with full-featured, well-designed programs to showcase the full power of the network. this is what we are all waiting for, and working towards. in practice, i suspect there will be at least one, and probably two as-yet-unnamed steps between homestead and metropolis: i’m open to suggestions for names (write to vinay[at]ethdev.com). features will be sensible checkpoints on the way: specific feature sets inside of mist would be my guess, but i’m still getting my head around that, so i expect we will cross those bridges after homestead is stood up. release step four: serenity there’s just one thing left to discuss: mining. proof of work implies the inefficient conversion of electricity into heat, ether and network stability, and we would quite like to not warm the atmosphere with our software more than is absolutely necessary. short of buying carbon offsets for every unit of ether mined (is that such a bad idea?), we need an algorithmic fix: the infamous proof of stake.  switching the network from proof of work to proof of stake is going to require a substantial switch, a transition process potentially much like the one between frontier and homestead. similar rollback measures may be required, although in all probability more sophisticated mechanisms will be deployed (e.g. running both mechanisms together, with proof of work dominant, and flagging any cases where proof of stake gives a different output.) this seems a long way out, but it’s not as far away as all that: the work is ongoing. proof of work is a brutal waste of computing power like democracy*, the worst system except all the others (*voluntarism etc. have yet to be tried at scale). freed from that constraint, the network should be faster, more efficient, easier for newcomers to get into, and more resistant to cartelization of mining capacity etc. this is probably going to be almost as big a step forwards as putting smart contracts into a block chain in the first place, by the time all is said and done. it is a ways out. it will be worth it.  timelines as you have seen since the ether sale, progress has been rapid and stable. code on the critical path is getting written, teams are effective and efficient, and over-all the organization is getting things done. reinventing the digital age is not easy, but somebody has to do it. right now that is us. we anticipate roughly one major announcement a month for the next few months, and then a delay while metropolis is prepared. there will also be devcon one, an opportunity to come, learn the practical business of building and shipping dapps, meet fellow developers, potential investors, and understand the likely shape of things to come. we will give you information about each release in more detail as each release approaches, but i want to give you the big overview of how this works and where we are going, fill in some of the gaps, highlight what is changing, both technically and in our communications and business partnership, and present you with an overview of what the summer is going to be like as we move down the path towards serenity, another world changing technology. i’m very glad to be part of this process. i’m a little at sea right now trying to wrap my head around the sheer scope of the project, and i’m hoping to actually visit a lot of the development teams over the summer to get the stories and put faces to names. this is a big, diverse project and, beyond the project itself, the launch of a new sociotechnical ecosystem. we are, after all, a platform effort: what’s really going to turn this into magic is you, and the things you build on top of the tools we’re all working so hard to ship. we are making tools for tool-makers. vinay signing off for now. more news soon! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle should ethereum be okay with enshrining more things in the protocol? 2023 sep 30 see all posts special thanks to justin drake, tina zhen and yoav weiss for feedback and review. from the start of the ethereum project, there was a strong philosophy of trying to make the core ethereum as simple as possible, and do as much as possible by building protocols on top. in the blockchain space, the "do it on l1" vs "focus on l2s" debate is typically thought of as being primarily about scaling, but in reality, similar issues exist for serving many kinds of ethereum users' needs: digital asset exchange, privacy, usernames, advanced cryptography, account safety, censorship resistance, frontrunning protection, and the list goes on. more recently, however, there has been some cautious interest in being willing to enshrine more of these features into the core ethereum protocol. this post will go into some of the philosophical reasoning behind the original minimal-enshrinement philosophy, as well as some more recent ways of thinking about some of these ideas. the goal will be to start to build toward a framework for better identifying possible targets where enshrining certain features in the protocol might be worth considering. early philosophy on protocol minimalism early on in the history of what was then called "ethereum 2.0", there was a strong desire to create a clean, simple and beautiful protocol that tried to do as little as possible itself, and left almost everything up to users to build on top. ideally, the protocol would just be a virtual machine, and verifying a block would just be a single virtual machine call. a very approximate reconstruction-from-memory of a whiteboard drawing gavin wood and i made back in early 2015, talking about what ethereum 2.0 would look like. the "state transition function" (the function that processes a block) would just be a single vm call, and all other logic would happen through contracts: a few system-level contracts, but mostly contracts provided by users. one really nice feature of this model is that even an entire hard fork could be described as a single transaction to the block processor contract, which would be approved through either offchain or onchain governance and then run with escalated permissions. these discussions back in 2015 particularly applied to two areas that were on our minds: account abstraction and scaling. in the case of scaling, the idea was to try to create a maximally abstracted form of scaling that would feel like a natural extension of the diagram above. a contract could make a call to a piece of data that was not stored by most ethereum nodes, and the protocol would detect that, and resolve the call through some kind of very generic scaled-computation functionality. from the virtual machine's point of view, the call would go off into some separate sub-system, and then some time later magically come back with the correct answer. this line of thinking was explored briefly, but soon abandoned, because we were too preoccupied with verifying that any kind of blockchain scaling was possible at all. though as we will see later, the combination of data availability sampling and zk-evms means that one possible future for ethereum scaling might actually look surprisingly close to that vision! for account abstraction, on the other hand, we knew from the start that some kind of implementation was possible, and so research immediately began to try to make something as close as possible to the purist starting point of "a transaction is just a call" into reality. there is a lot of boilerplate code that occurs in between processing a transaction and making the actual underlying evm call out of the sender address, and a lot more boilerplate that comes after. how do we reduce this code to as close to nothing as possible? one of the major pieces of code in here is validate_transaction(state, tx), which does things like checking that the nonce and signature of the transaction are correct. the practical goal of account abstraction was, from the start, to allow the user to replace basic nonce-incrementing and ecdsa validation with their own validation logic, so that users could more easily use things like social recovery and multisig wallets. hence, finding a way to rearchitect apply_transaction into just being a simple evm call was not simply a "make the code clean for the sake of making the code clean" task; rather, it was about moving the logic into the user's account code, to give users that needed flexibility. however, the insistence on trying to make apply_transaction contain as little enshrined logic as possible ended up introducing a lot of challenges. to see why, let us zoom in on one of the earliest account abstraction proposals, eip 86: specification if block.number >= metropolis_fork_blknum, then: 1. if the signature of a transaction is (0, 0, 0) (ie. v = r = s = 0), then treat it as valid and set the sender address to 2**160 1 2. set the address of any contract created through a creation transaction to equal sha3(0 + init code) % 2**160, where + represents concatenation, replacing the earlier address formula of sha3(rlp.encode([sender, nonce])) 3. create a new opcode at 0xfb, create_p2sh, which sets the creation address to sha3(sender + init code) % 2**160. if a contract at that address already exists, fails and returns 0 as if the init code had run out of gas. basically, if the signature is set to (0, 0, 0), then a transaction really does become "just a call". the account itself would be responsible for having code that parses the transaction, extracts and verifies the signature and nonce, and pays fees; see here for an early example version of that code, and see here for the very similar validate_transaction code that this account code would be replacing. in exchange for this simplicity at protocol layer, miners (or, today, block proposers) gain the additional responsibility of running extra logic for only accepting and forwarding transactions that go to accounts whose code is set up to actually pay fees. what is that logic? well, honestly eip-86 did not think too hard about it: note that miners would need to have a strategy for accepting these transactions. this strategy would need to be very discriminating, because otherwise they run the risk of accepting transactions ) for the validate_transaction code that this pre-account code would be replacingthat do not pay them any fees, and possibly even transactions that have no effect (eg. because the transaction was already included and so the nonce is no longer current). one simple approach is to have a whitelist for the codehash of accounts that they accept transactions being sent to; approved code would include logic that pays miners transaction fees. however, this is arguably too restrictive; a looser but still effective strategy would be to accept any code that fits the same general format as the above, consuming only a limited amount of gas to perform nonce and signature checks and having a guarantee that transaction fees will be paid to the miner. another strategy is to, alongside other approaches, try to process any transaction that asks for less than 250,000 gas, and include it only if the miner's balance is appropriately higher after executing the transaction than before it. if eip-86 had been included as-is, it would have reduced the complexity of the evm, at the cost of massively increasing the complexity of other parts of the ethereum stack, requiring essentially the exact same code to be written in other places, in addition to introducing entirely new classes of weirdness such as the possibility that the same transaction with the same hash might appear multiple times in the chain, not to mention the multi-invalidation problem. the multi-invalidation problem in account abstraction. one transaction getting included on chain could invalidate thousands of other transactions in the mempool, making the mempool easy to cheaply flood. acccount abstraction evolved in stages from there. eip-86 became eip-208, which later became this ethresear.ch post on "tradeoffs in account abstraction proposals", which then became this ethresear.ch post half a year later. eventually, out of all this, came the actually somewhat-workable eip-2938. eip-2938, however, was not minimalistic at all. the eip includes: a new transaction type three new transaction-wide global variables two new opcodes, including the highly unwieldy paygas opcode that handles gas price and gas limit checking, being an evm execution breakpoint, and temporarily storing eth for fee payments all at once. a set of complex mining and rebroadcasting strategies, including a list of banned opcodes for the validation phase of a transaction in order to get account abstraction off the ground without involving ethereum core developers who were busy on heroic efforts optimizing the ethereum clients and implementing the merge, eip-2938 eventually was rearchitected into the entirely extra-protocol erc-4337. erc-4337. it really does rely entirely on evm calls for everything! because it's an erc, it does not require a hard fork, and technically lives "outside of the ethereum protocol". so.... problem solved? well, as it turns out, not quite. the current medium-term roadmap for erc-4337 actually does involve eventually turning large parts of erc-4337 into a series of protocol features, and it's a useful instructive example to see the reasons why this path is being considered. enshrining erc-4337 there have been a few key reasons discussed for eventually bringing erc-4337 back into the protocol: gas efficiency: anything done inside the evm incurs some level of virtual machine overhead, including inefficiency in how it uses gas-expensive features like storage slots. currently, these extra inefficiencies add up to at least ~20,000 gas, and often more. pushing these components into the protocol is the easiest way to remove these issues. code bug risk: if the erc-4337 "entry point contract" has a sufficiently terrible bug, all erc-4337-compatible wallets could see all of their funds drained. replacing the contract with an in-protocol functionality creates an implied responsibility to fix code bugs with a hard fork, which removes funds-draining risk for users. support for evm opcodes like tx.origin. erc-4337, by itself, makes tx.origin return the address of the "bundler" that packaged up a set of user operations into a transaction. native account abstraction could fix this, by making tx.origin point to the actual account sending the transaction, making it work the same way as for eoas. censorship resistance: one of the challenges with proposer/builder separation is that it becomes easier to censor individual transactions. in a world where individual transactions are legible to the ethereum protocol, this problem can be greatly mitigated with inclusion lists, which allow proposers to specify a list of transactions that must be included within the next two slots in almost all cases. but the extra-protocol erc-4337 wraps "user operations" inside a single transaction, making user operations opaque to the ethereum protocol; hence, ethereum-protocol-provided inclusion lists would not be able to provide censorship resistance to erc-4337 user operations. enshrining erc-4337, and making user operations a "proper" transaction type, would solve this problem. it's worth zooming into the gas efficiency issue further. in its current form, erc-4337 is significantly more expensive than a "basic" ethereum transaction: the transaction costs 21,000 gas, whereas erc-4337 costs ~42,000 gas. this doc lists some of the reasons why: need to pay lots of individual storage read/write costs, which in the case of eoas get bundled into a single 21000 gas payment: editing the storage slot that contains pubkey+nonce (~5000) useroperation calldata costs (~4500, reducible to ~2500 with compression) ecrecover (~3000) warming the wallet itself (~2600) warming the recipient account (~2600) transferring eth to the recipient account (~9000) editing storage to pay fees (~5000) access the storage slot containing the proxy (~2100) and then the proxy itself (~2600) on top of the above storage read/write costs, the contract needs to do "business logic" (unpacking the useroperation, hashing it, shuffling variables, etc) that eoa transactions have handled "for free" by the ethereum protocol need to expend gas to pay for logs (eoas don't issue logs) one-time contract creation costs (~32000 base, plus 200 gas per code byte in the proxy, plus 20000 to set the proxy address) theoretically, it should be possible to massage the evm gas cost system until the in-protocol costs and the extra-protocol costs for accessing storage match; there is no reason why transferring eth needs to cost 9000 gas when other kinds of storage-editing operations are much cheaper. and indeed, two eips ([1] [2]) related to the upcoming verkle tree transition actually try to do that. but even if we do that, there is one huge reason why enshrined protocol features are going to inevitably be significantly cheaper than evm code, no matter how efficient the evm becomes: enshrined code does not need to pay gas for being pre-loaded. fully functional erc-4337 wallets are big. this implementation, compiled and put on chain, takes up ~12,800 bytes. of course, you can deploy that big piece of code once, and use delegatecall to allow each individual wallet to call into it, but that code still needs to be accessed in each block that uses it. under the verkle tree gas costs eip, 12,800 bytes would make up 413 chunks, and accessing those chunks would require paying 2x witness_branch_cost (3,800 gas total) and 413x witness_chunk_cost (82,600 gas total). and this does not even begin to mention the erc-4337 entry-point itself, with 23,689 bytes onchain in version 0.6.0 (under the verkle tree eip rules, ~158,700 gas to load). this leads to a problem: the gas costs of actually accessing this code would have to be split among transactions somehow. the current approach that erc-4337 uses is not great: the first transaction in a bundle eats up one-time storage/code reading costs, making it much more expensive than the rest of the transactions. enshrinement in-protocol would allow these commonly-shared libraries to simply be part of the protocol, accessible to all with no fees. what can we learn from this example about when to enshrine things more generally? in this example, we saw a few different rationales for enshrining aspects of account abstraction in the protocol. "move complexity to the edges" market-based approaches break down the most when there are high fixed costs. and indeed, the long term account abstraction roadmap looks like it's going to have lots of fixed costs per block. 244,100 gas for loading standardized wallet code is one thing; but aggregation (see my presentation from this summer for more details) potentially adds hundreds of thousands more gas for zk-snark validation plus onchain costs for proof verification. there isn't a way to charge users for these costs without introducing lots of market inefficiencies, whereas making some of these functionalities into protocol features accessible to all with no fees cleanly solves that problem. community-wide response to code bugs. if some set of pieces of code are used by all users, or a very wide class of users, then it often makes more sense for the blockchain community to take on itself the responsibility to hard-fork to fix any bugs that arise. erc-4337 introduced a large amount of globally shared code, and in the long term it makes more sense for bugs in that code to be fixed by hard forks than to lead to users losing a large amount of eth. sometimes, a stronger form of a feature can be implemented by directly taking advantage of the powers of the protocol. the key example here is in-protocol censorship resistance features like inclusion lists: in-protocol inclusion lists can do a better job of guaranteeing censorship resistance than extra-protocol approaches, in order for user-level operations to actually benefit from in-protocol inclusion lists, individual user-level operations need to be "legible" to the protocol. another lesser-known example is that 2017-era ethereum proof of stake designs had account abstraction for staking keys, and this was abandoned in favor of enshrining bls because bls supported an "aggregation" mechanism, which would have to be implemented at protocol and network level, that could make handling a very large number of signatures much more efficient. but it is important to remember that even enshrined in-protocol account abstraction is still a massive "de-enshrinement" compared to the status quo. today, top-level ethereum transactions can only be initiated from externally owned accounts (eoas) which use a single secp256k1 elliptic curve signature for verification. account abstraction de-enshrines this, and leaves verification conditions open for users to define. and so, in this story about account abstraction, we also saw the biggest argument against enshrinement: being flexible to diverse users' needs. let us try to fill in the story further, by looking at a few other examples of features that have recently been considered for enshrinement. we'll particularly focus on: zk-evms, proposer-builder separation, private mempools, liquid staking and new precompiles. enshrining zk-evms let us switch focus to another potential target for enshrining into the ethereum protocol: zk-evms. currently, we have a large number of zk-rollups that all have to write fairly similar code to verify execution of ethereum-like blocks inside a zk-snark. there is a pretty diverse ecosystem of independent implementations: the pse zk-evm, kakarot, the polygon zk-evm, linea, zeth, and the list goes on. one of the recent controversies in the evm zk-rollup space has to do with how to deal with the possibility of bugs in the zk-code. currently, all of these systems that are live have some form of "security council" mechanism that can override the proving system in case of a bug. in this post last year, i tried to create a standardized framework to encourage projects to be clear about what level of trust they put in the proving system and what level in the security council, and move toward giving less and less powers to the security council over time. in the medium term, rollups could rely on multiple proving systems, and the security council would only have any power at all in the extreme case where two different proving systems disagree with each other. however, there is a sense in which some of this work feels superfluous. we already have the ethereum base layer, which has an evm, and we already have a working mechanism for dealing with bugs in implementations: if there's a bug, the clients that have the bug update to fix the bug, and the chain goes on. blocks that appeared finalized from the perspective of a buggy client would end up no-longer-finalized, but at least we would not see users losing funds. similarly, if a rollup just wants to be and remain evm-equivalent, it feels wrong that they need to implement their own governance to keep changing their internal zk-evm rules to match upgrades to the ethereum base layer, when ultimately they're building on top of the ethereum base layer itself, which knows when it's being upgraded and to what new rules. since these l2 zk-evms are basically using the exact same evm as ethereum, can't we somehow make "verify evm execution in zk" into a protocol feature, and deal with exceptional situations like bugs and upgrades by just applying ethereum's social consensus, the same way we already do for base-layer evm execution itself? this is an important and challenging topic. there are a few nuances: we want to be compatible with ethereum's multi-client philosophy. this means that we want to allow different clients to use different proving systems. this in turn implies that for any evm execution that gets proven with one zk-snark system, we want a guarantee that the underlying data is available, so that proofs can be generated for other zk-snark systems. while the tech is immature, we probably want auditability. in practice, this means the exact same thing: if any execution gets proven, we want the underlying data to be available, so that if anything goes wrong, users and developers can inspect it. we need much faster proving times, so that if one type of proof is made, other types of proof can be generated quickly enough that other clients can validate them. one could get around this by making a precompile that has an asynchronous response after some time window longer than a slot (eg. 3 hours), but this adds complexity. we want to support not just copies of the evm, but also "almost-evms". part of the attraction of l2s is the ability to innovate on the execution layer, and make extensions to the evm. if a given l2's vm differs from the evm only a little bit, it would be nice if the l2 could still use a native in-protocol zk-evm for the parts that are identical to the evm, and only rely on their own code for the parts that are different. this could be done by designing the zk-evm precompile in such a way that it allows the caller to specify a bitfield or list of opcodes or addresses that get handled by an externally supplied table instead of the evm itself. we could also make gas costs open to customization to a limited extent. one likely topic of contention with data availability in a native zk-evm is statefulness. zk-evms are much more data-efficient if they do not have to carry "witness" data. that is, if a particular piece of data was already read or written in some previous block, we can simply assume that provers have access to it, and we don't have to make it available again. this goes beyond not re-loading storage and code; it turns out that if a rollup properly compresses data, the compression being stateful allows for up to 3x data savings compared to the compression being stateless. this means that for a zk-evm precompile, we have two options: the precompile requires all data to be available in the same block. this means that provers can be stateless, but it also means that zk-rollups using such a precompile become much more expensive than rollups using custom code. the precompile allows pointers to data used or generated by previous executions. this allows zk-rollups to be near-optimal, but it's more complicated and introduces a new kind of state that has to be stored by provers. what lessons can we take away from this? there is a pretty good argument to enshrine zk-evm validation somehow: rollups are already building their own custom versions of it, and it feels wrong that ethereum is willing to put the weight of its multiple implementations and off-chain social consensus behind evm execution on l1, but l2s doing the exact same work have to instead implement complicated gadgets involving security councils. but on the other hand, there is a big devil in the details: there are different versions of an enshrined zk-evm that have different costs and benefits. the stateful vs stateless divide only scratches the surface; attempting to support "almost-evms" that have custom code proven by other systems will likely reveal an even larger design space. hence, enshrining zk-evms presents both promise and challenges. enshrining proposer-builder separation (epbs) the rise of mev has made block production into an economies-of-scale-heavy activity, with sophisticated actors being able to produce blocks that generate much more revenue than default algorithms that simply watch the mempool for transactions and include them. the ethereum community has so far attempted to deal with this by using extra-protocol proposer-builder separation schemes like mev-boost, which allow regular validators ("proposers") to outsource block building to specialized actors ("builders"). however, mev-boost carries a trust assumption in a new category of actor, called a relay. for the past two years, there have been many proposals to create "enshrined pbs". what is the benefit of this? in this case, the answer is pretty simple: the pbs that can be built by directly using the powers of the protocol is simply stronger (in the sense of having weaker trust assumptions) than the pbs that can be built without them. it's a similar case to the case for enshrining in-protocol price oracles though, in that situation, there is also a strong counterargument. enshrining private mempools when a user sends a transaction, that transaction becomes immediately public and visible to all, even before it gets included on chain. this makes users of many applications vulnerable to economic attacks such as frontrunning: if a user makes a large trade on eg. uniswap, an attacker could put in a transaction right before them, increasing the price at which they buy, and collecting an arbitrage profit. recently, there has been a number of projects specializing in creating "private mempools" (or "encrypted mempools"), which keep users' transactions encrypted until the moment they get irreversibly accepted into a block. the problem is, however, that schemes like this require a particular kind of encryption: to prevent users from flooding the system and frontrunning the decryption process itself, the encryption must auto-decrypt once the transaction actually does get irreversibly accepted. to implement such a form of encryption, there are various different technologies with different tradeoffs, described well in this post by jon charbonneau (and this video and slides): encryption to a centralized operator, eg. flashbots protect. time-lock encryption, a form of encryption which can be decrypted by anyone after a certain number of sequential computational steps, which cannot be parallelized. threshold encryption, trusting an honest majority committee to decrypt the data. see the shutterized beacon chain concept for a concrete proposal. trusted hardware such as sgx. unfortunately, each of these have varying weaknesses. a centralized operator is not acceptable for inclusion in-protocol for obvious reasons. traditional time lock encryption is too expensive to run across thousands of transactions in a public mempool. a more powerful primitive called delay encryption allows efficient decryption of an unlimited number of messages, but it's hard to construct in practice, and attacks on existing constructions still sometimes get discovered. much like with hash functions, we'll likely need a period of more years of research and analysis before delay encryption becomes sufficiently mature. threshold encryption requires trusting a majority to not collude, in a setting where they can collude undetectably (unlike 51% attacks, where it's immediately obvious who participated). sgx creates a dependency on a single trusted manufacturer. while for each solution, there is some subset of users that is comfortable trusting it, there is no single solution that is trusted enough that it can practically be accepted into layer 1. hence, enshrining anti-frontrunning at layer 1 seems like a difficult proposition at least until delay encrypted is perfected or there is some other technological breakthrough, even while it's a valuable enough functionality that lots of application solutions will already emerge. enshrining liquid staking a common demand among ethereum defi users is the ability to use their eth for staking and as collateral in other applications at the same time. another common demand is simply for convenience: users want to be able to stake without the complexity of running a node and keeping it online all the time (and protecting their now-online staking keys). by far the simplest possible "interface" for staking, which satisfies both of these needs, is just an erc20 token: convert your eth into "staked eth", hold it, and then later convert back. and indeed, liquid staking providers such as lido and rocketpool have emerged to do just that. however, liquid staking has some natural centralizing mechanics at play: people naturally go into the biggest version of staked eth because it's most familiar and most liquid (and most well-supported by applications, who in turn support it because it's more familiar and because it's the one the most users will have heard of). each version of staked eth needs to have some mechanism determining who can be the underlying node operators. it can't be unrestricted, because then attackers would join and amplify their attacks with users' funds. currently, the top two are lido, which has a dao whitelisting node operators, and rocket pool, which allows anyone to run a node if they put down 8 eth (ie. 1/4 of the capital) as a deposit. these two approaches have different risks: the rocket pool approach allows attackers to 51% attack the network, and force users to pay most of the costs. with the dao approach, if a single such staking token dominates, that leads to a single, potentially attackable governance gadget controlling a very large portion of all ethereum validators. to the credit of protocols like lido, they have implemented safeguards against this, but one layer of defense may not be enough. in the short term, one option is to socially encourage ecosystem participants to use a diversity of liquid staking providers, to reduce the chance that any single one becomes too large to be a systemic risk. in the longer term, however, this is an unstable equilibrium, and there is peril in relying too much on moralistic pressure to solve problems. one natural question arises: might it make sense to enshrine some kind of in-protocol functionality to make liquid staking less centralizing? here, the key question is: what kind of in-protocol functionality? simply creating an in-protocol fungible "staked eth" token has the problem that it would have to either have an enshrined ethereum-wide governance to choose who runs the nodes, or be open-entry, turning it into a vehicle for attackers. one interesting idea is dankrad feist's writings on liquid staking maximalism. first, we bite the bullet that if ethereum gets 51% attacked, only perhaps 5% of the attacking eth gets slashed. this is a reasonable tradeoff; right now there is over 26 million eth being staked, and a cost of attack of 1/3 of that (~8 million eth) is way overkill, especially considering how many kinds of "outside-the-model" attacks can be pulled off for much less. indeed, a similar tradeoff has already been explored in the "super-committee" proposal for implementing single-slot finality. if we accept that only 5% of attacking eth gets slashed, then over 90% of staked eth would be invulnerable to slashing, and so 90% of staked eth could be put into an in-protocol fungible liquid staking token that can then be used by other applications. this path is interesting. but it still leaves open the question: what is the specific thing that would get enshrined? rocketpool already works in a way very similar to this: each node operator puts up some capital, and liquid stakers put up the rest. we could simply tweak a few constants, bounding the maximum slashing penalty to eg. 2 eth, and rocket pool's existing reth would become risk-free. there are other clever things that we can do with simple protocol tweaks. for example, imagine that we want a system where there are two "tiers" of staking: node operators (high collateral requirement) and depositors (no minimum, can join and leave any time), but we still want to guard against node operator centralization by giving a randomly-sampled committee of depositors powers like suggesting lists of transactions that have to be included (for anti-censorship reasons), controlling the fork choice during an inactivity leak, or needing to sign off on blocks. this could be done in a mostly-out-of-protocol way, by tweaking the protocol to require each validator to provide (i) a regular staking key, and (ii) an eth address that can be called to output a secondary staking key during each slot. the protocol would give powers to these two keys, but the mechanism for choosing the second key in each slot could be left to staking pool protocols. it may still be better to enshrine some things outright, but it's valuable to note that this "enshrine some things, leave other things to users" design space exists. enshrining more precompiles precompiles (or "precompiled contracts") are ethereum contracts that implement complex cryptographic operations, whose logic is natively implemented in client code, instead of evm smart contract code. precompiles were a compromise adopted at the beginning of ethereum's development: because the overhead of a vm is too much for certain kinds of very complex and highly specialized code, we can implement a few key operations valuable to important kinds of applications in native code to make them faster. today, this basically includes a few specific hash functions and elliptic curve operations. there is currently a push to add a precompile for secp256r1, an elliptic curve slightly different from the secp256k1 used for basic ethereum accounts, because it is well-supported by trusted hardware modules and thus widespread use of it could improve wallet security. in recent years, there have also been pushes to add precompiles for bls-12-377, bw6-761, generalized pairings and other features. the counterargument to these requests for more precompiles is that many of the precompiles that have been added before (eg. ripemd and blake) have ended up gotten used much less than anticipated, and we should learn from that. instead of adding more precompiles for specific operations, we should perhaps focus on a more moderate approach based on ideas like evm-max and the dormant-but-always-revivable simd proposal, which would allow evm implementations to execute wide classes of code less expensively. perhaps even existing little-used precompiles could be removed and replaced with (unavoidably less efficient) evm code implementations of the same function. that said, it is still possible that there are specific cryptographic operations that are valuable enough to accelerate that it makes sense to add them as precompiles. what do we learn from all this? the desire to enshrine as little as possible is understandable and good; it hails from the unix philosophy tradition of creating software that is minimalist and can be easily adapted to different needs by its users, avoiding the curses of software bloat. however, blockchains are not personal-computing operating systems; they are social systems. this means that there are rationales for enshrining certain features in the protocol that go beyond the rationales that exist in a purely personal-computing context. in many cases, these other examples re-capped similar lessons to what we saw in account abstraction. but there are also a few new lessons that have been learned as well: enshrining features can help avoid centralization risks in other areas of the stack. often, keeping the base protocol minimal and simple pushes the complexity to some outside-the-protocol ecosystem. from a unix philosophy perspective, this is good. sometimes, however, there are risks that that outside-the-protocol ecosystem will centralize, often (but not just) because of high fixed costs. enshrining can sometimes decrease de-facto centralization. enshrining too much can over-extend the trust and governance load of the protocol. this is the topic of this earlier post about not overloading ethereum's consensus: if enshrining a particular feature weakens the trust model, and makes ethereum as a whole much more "subjective", that weakens ethereum's credible neutrality. in those cases, it's better to leave that particular feature as a mechanism on top of ethereum, and not try to bring it inside ethereum itself. here, encrypted mempools are the best example of something that may be a bit too difficult to enshrine, at least until/unless delay encryption technology improves. enshrining too much can over-complicate the protocol. protocol complexity is a systemic risk, and adding too many features in-protocol increases that risk. precompiles are the best example of this. enshrining can backfire in the long term, as users' needs are unpredictable. a feature that many people think is important and will be used by many users may well turn out not to be used much in practice. additionally, the liquid staking, zk-evm and precompile cases show the possibility of a middle road: minimal viable enshrinement. rather than enshrining an entire functionality, the protocol could enshrine a specific piece that solves the key challenges with making that functionality easy to implement, without being too opinionated or narrowly focused. examples of this include: rather than enshrining a full liquid staking system, changing staking penalty rules to make trustless liquid staking more viable rather than enshrining more precompiles, enshrine evm-max and/or simd to make a wider class of operations simpler to implement efficiently rather than enshrining the whole concept of rollups, we could simply enshrine evm verification. we can extend our diagram from earlier in the post as follows: sometimes, it may even make sense to de-enshrine a few things. de-enshrining little-used precompiles is one example. account abstraction as a whole, as mentioned earlier, is also a significant form of de-enshrinement. if we want to support backwards-compatibility for existing users, then the mechanism may actually be surprisingly similar to that for de-enshrining precompiles: one of the proposals is eip-5003, which would allow eoas to convert their account in-place into a contract that has the same (or better) functionality. what features should be brought into the protocol and what features should be left to other layers of the ecosystem is a complicated tradeoff, and we should expect the tradeoff to continue to evolve over time as our understanding of users' needs and our suite of available ideas and technologies continues to improve. dark mode toggle review of gitcoin quadratic funding round 4 2020 jan 28 see all posts round 4 of gitcoin grants quadratic funding has just completed, and here are the results: the main distinction between round 3 and round 4 was that while round 3 had only one category, with mostly tech projects and a few outliers such as ethhub, in round 4 there were two separate categories, one with a $125,000 matching pool for tech projects, and the other with a $75,000 matching pool for "media" projects. media could include documentation, translation, community activities, news reporting, theoretically pretty much anything in that category. and while the tech section went about largely without incident, in the new media section the results proved to be much more interesting than i could have possibly imagined, shedding a new light on deep questions in institutional design and political science. tech: quadratic funding worked great as usual in the tech section, the main changes that we see compared to round 3 are (i) the rise of tornado cash and (ii) the decline in importance of eth2 clients and the rise of "utility applications" of various forms. tornado cash is a trustless smart contract-based ethereum mixer. it became popular quickly in recent months, as the ethereum community was swept by worries about the blockchain's current low levels of privacy and wanted solutions. tornado cash amassed an incredible $31,200. if they continue receiving such an amount every two months then this would allow them to pay two people $7,800 per month each meaning that the hoped-for milestone of seeing the first "quadratic freelancer" may have already been reached! the other major winners included tools like dappnode, a software package to help people run nodes, sablier, a payment streaming service, and defizap, which makes defi services easy to use. the gitcoin sustainability fund got over $13,000, conclusively resolving my complaint from last round that they were under-supported. all in all, valuable grants for valuable projects that provide services that the community genuinely needs. we can see one major shift this round compared to the previous rounds. whereas in previous rounds, the grants went largely to projects like eth2 clients that were already well-supported, this time the largest grants shifted toward having a different focus from the grants given by the ethereum foundation. the ef has not given grants to tornado.cash, and generally limits its grants to application-specific tools, uniswap being a notable exception. the gitcoin grants quadratic fund, on the other hand, is supporting defizap, sablier, and many other tools that are valuable to the community. this is arguably a positive development, as it allows gitcoin grants and the ethereum foundation to complement each other rather than focusing on the same things. the one proposed change to the quadratic funding implementation for tech that i would favor is a user interface change, that makes it easier for users to commit funds for multiple rounds. this would increase the stability of contributions, thereby increasing the stability of projects' income very important if we want "quadratic freelancer" to actually be a viable job category! media: the first quadratic twitter freelancer now, we get to the new media section. in the first few days of the round, the leading recipient of the grants was "@antiprosynth twitter account activity": an ethereum community member who is very active on twitter promoting ethereum and refuting misinformation from bitcoin maximalists, asking for help from the gitcoin qf crowd to.... fund his tweeting activities. at its peak, the projected matching going to @antiprosynth exceeded $20,000. this naturally proved to be controversial, with many criticizing this move and questioning whether or not a twitter account is a legitimate public good: on the surface, it does indeed seem like someone getting paid $20,000 for operating a twitter account is ridiculous. but it's worth digging in and questioning exactly what, if anything, is actually wrong with this outcome. after all, maybe this is what effective marketing in 2020 actually looks like, and it's our expectations that need to adapt. there are two main objections that i heard, and both lead to interesting criticisms of quadratic funding in its current implementation. first, there was criticism of overpayment. twittering is a fairly "trivial" activity; it does not require that much work, lots of people do it for free, and it doesn't provide nearly as much long-term value as something more substantive like ethhub or the zero knowledge podcast. hence, it feels wrong to pay a full-time salary for it. examples of @antiprosynth's recent tweets if we accept the metaphor of quadratic funding as being like a market for public goods, then one could simply extend the metaphor, and reply to the concern with the usual free-market argument. people voluntarily paid their own money to support @antiprosynth's twitter activity, and that itself signals that it's valuable. why should we trust you with your mere words and protestations over a costly signal of real money on the table from dozens of people? the most plausible answer is actually quite similar to one that you often hear in discussions about financial markets: markets can give skewed results when you can express an opinion in favor of something but cannot express an opinion against it. when short selling is not possible, financial markets are often much more inefficient, because instead of reflecting the average opinion on an asset's true value, a market may instead reflect the inflated expectations of an asset's few rabid supporters. in this version of quadratic funding, there too is an asymmetry, as you can donate in support of a project but you cannot donate to oppose it. might this be the root of the problem? one can go further and ask, why might overpayment happen to this particular project, and not others? i have heard a common answer: twitter accounts already have a high exposure. a client development team like nethermind does not gain much publicity through their work directly, so they need to separately market themselves, whereas a twitter account's "work" is self-marketing by its very nature. furthermore, the most prominent twitterers get quadratically more matching out of their exposure, amplifying their outsized advantage further a problem i alluded to in my review of round 3. interestingly, in the case of vanilla quadratic voting there was an argument made by glen weyl for why economies-of-scale effects of traditional voting, such as duverger's law, don't apply to quadratic voting: a project becoming more prominent increases the incentive to give it both positive and negative votes, so on net the effects cancel out. but notice once again, that this argument relies on negative votes being a possibility. good for the tribe, but is it good for the world? the particular story of @antiprosynth had what is in my opinion a happy ending: over the next ten days, more contributions came in to other candidates, and @antiprosynth's match reduced to $11,316, still a respectably high amount but on par with ethhub and below week in ethereum. however, even a quadratic matching grant of this size still raises to the next criticism: is twittering a public good or public bad anyway? traditionally, public goods of the type that gitcoin grants quadratic funding is trying to support were selected and funded by governments. the motivation of @antiprosynth's tweets is "aggregating ethereum-related news, fighting information asymmetry and fine-tuning/signaling a consistent narrative for ethereum (and eth)": essentially, fighting the good fight against anti-ethereum misinformation by bitcoin maximalists. and, lo and behold, governments too have a rich history of sponsoring social media participants to argue on their behalf. and it seems likely that most of these governments see themselves as "fighting the good fight against anti-[x] misinformation by [y] {extremists, imperialists, totalitarians}", just as the ethereum community feels a need to fight the good fight against maximalist trolls. from the inside view of each individual country (and in our case the ethereum community) organized social media participation seems to be a clear public good (ignoring the possibility of blowback effects, which are real and important). but from the outside view of the entire world, it can be viewed as a zero-sum game. this is actually a common pattern to see in politics, and indeed there are many instances of larger-scale coordination that are precisely intended to undermine smaller-scale coordination that is seen as "good for the tribe but bad for the world": antitrust law, free trade agreements, state-level pre-emption of local zoning codes, anti-militarization agreements... the list goes on. a broad environment where public subsidies are generally viewed suspiciously also does quite a good job of limiting many kinds of malign local coordination. but as public goods become more important, and we discover better and better ways for communities to coordinate on producing them, that strategy's efficacy becomes more limited, and properly grappling with these discrepancies between what is good for the tribe and what is good for the world becomes more important. that said, internet marketing and debate is not a zero-sum game, and there are plenty of ways to engage in internet marketing and debate that are good for the world. internet debate in general serves to help the public learn what things are true, what things are not true, what causes to support, and what causes to oppose. some tactics are clearly not truth-favoring, but other tactics are quite truth-favoring. some tactics are clearly offensive, but others are defensive. and in the ethereum community, there is widespread sentiment that there is not enough resources going into marketing of some kind, and i personally agree with this sentiment. what kind of marketing is positive-sum (good for tribe and good for world) and what kind of marketing is zero-sum (good for tribe but bad for world) is another question, and one that's worth the community debating. i naturally hope that the ethereum community continues to value maintaining a moral high ground. regarding the case of @antiprosynth himself, i cannot find any tactics that i would classify as bad-for-world, especially when compared to outright misinformation ("it's impossible to run a full node") that we often see used against ethereum but i am pro-ethereum and hence biased, hence the need to be careful. universal mechanisms, particular goals the story has another plot twist, which reveals yet another feature (or bug?) or quadratic funding. quadratic funding was originally described as "formal rules for a society neutral among communities", the intention being to use it at a very large, potentially even global, scale. anyone can participate as a project or as a participant, and projects that support public goods that are good for any "public" would be supported. in the case of gitcoin grants, however, the matching funds are coming from ethereum organizations, and so there is an expectation that the system is there to support ethereum projects. but there is nothing in the rules of quadratic funding that privileges ethereum projects and prevents, say, ethereum classic projects from seeking funding using the same platform! and, of course, this is exactly what happened: so now the result is, $24 of funding from ethereum organizations will be going toward supporting an ethereum classic promoter's twitter activity. to give people outside of the crypto space a feeling for what this is like, imagine the usa holding a quadratic funding raise, using government funding to match donations, and the result is that some of the funding goes to someone explicitly planning to use the money to talk on twitter about how great russia is (or vice versa). the matching funds are coming from ethereum sources, and there's an implied expectation that the funds should support ethereum, but nothing actually prevents, or even discourages, non-ethereum projects from organizing to get a share of the matched funds on the platform! solutions there are two solutions to these problems. one is to modify the quadratic funding mechanism to support negative votes in addition to positive votes. the mathematical theory behind quadratic voting already implies that it is the "right thing" to do to allow such a possibility (every positive number has a negative square root as well as a positive square root). on the other hand, there are social concerns that allowing for negative voting would cause more animosity and lead to other kinds of harms. after all, mob mentality is at its worst when it is against something rather than for something. hence, it's my view that it's not certain that allowing negative contributions will work out well, but there is enough evidence that it might that it is definitely worth trying out in a future round. the second solution is to use two separate mechanisms for identifying relative goodness of good projects and for screening out bad projects. for example, one could use a challenge mechanism followed by a majority eth coin vote, or even at first just a centralized appointed board, to screen out bad projects, and then use quadratic funding as before to choose between good projects. this is less mathematically elegant, but it would solve the problem, and it would at the same time provide an opportunity to mix in a separate mechanism to ensure that chosen projects benefit ethereum specifically. but even if we adopt the first solution, defining boundaries for the quadratic funding itself may also be a good idea. there is intellectual precedent for this. in elinor ostrom's eight principles for governing the commons, defining clear boundaries about who has the right to access the commons is the first one. without clear boundaries, ostrom writes, "local appropriators face the risk that any benefits they produce by their efforts will be reaped by others who have not contributed to those efforts." in the case of gitcoin grants quadratic funding, one possibility would be to set the maximum matching coefficient for any pair of users to be proportional to the geometric average of their eth holdings, using that as a proxy for measuring membership in the ethereum community (note that this avoids being plutocratic because 1000 users with 1 eth each would have a maximum matching of \(\approx k * 500,000\) eth, whereas 2 users with 500 eth each would only have a maximum matching of \(k * 1,000\) eth). collusion another issue that came to the forefront this round was the issue of collusion. the math behind quadratic funding, which compensates for tragedies of the commons by magnifying individual contributions based on the total number and size of other contributions to the same project, only works if there is an actual tragedy of the commons limiting natural donations to the project. if there is a "quid pro quo", where people get something individually in exchange for their contributions, the mechanism can easily over-compensate. the long-run solution to this is something like maci, a cryptographic system that ensures that contributors have no way to prove their contributions to third parties, so any such collusion would have to be done by honor system. in the short run, however, the rules and enforcement has not yet been set, and this has led to vigorous debate about what kinds of quid pro quo are legitimate: [update 2020.01.29: the above was ultimately a result of a miscommunication from gitcoin; a member of the gitcoin team had okayed richard burton's proposal to give rewards to donors without realizing the implications. so richard himself is blameless here; though the broader point that we underestimated the need for explicit guidance about what kinds of quid pro quos are acceptable is very much real.] currently, the position is that quid pro quos are disallowed, though there is a more nuanced feeling that informal social quid pro quos ("thank yous" of different forms) are okay, whereas formal and especially monetary or product rewards are a no-no. this seems like a reasonable approach, though it does put gitcoin further into the uncomfortable position of being a central arbiter, compromising credible neutrality somewhat. one positive byproduct of this whole discussion is that it has led to much more awareness in the ethereum community of what actually is a public good (as opposed to a "private good" or a "club good"), and more generally brought public goods much further into the public discourse. conclusions whereas round 3 was the first round with enough participants to have any kind of interesting effects, round 4 felt like a true "coming-out party" for the cause of decentralized public goods funding. the round attracted a large amount of attention from the community, and even from outside actors such as the bitcoin community. it is part of a broader trend in the last few months where public goods funding has become a dominant part of the crypto community discourse. along with this, we have also seen much more discussion of strategies about long-term sources of funding for quadratic matching pools of larger sizes. discussions about funding will be important going forward: donations from large ethereum organizations are enough to sustain quadratic matching at its current scale, but not enough to allow it to grow much further, to the point where we can have hundreds of quadratic freelancers instead of about five. at those scales, sources of funding for ethereum public goods must rely on network effect lockin to some extent, or else they will have little more staying power than individual donations, but there are strong reasons not to embed these funding sources too deeply into ethereum (eg. into the protocol itself, a la the recent bch proposal), to avoid risking the protocol's neutrality. approaches based on capturing transaction fees at layer 2 are surprisingly viable: currently, there are about $50,000-100,000 per day (~$18-35m per year) of transaction fees happening on ethereum, roughly equal to the entire budget of the ethereum foundation. and there is evidence that miner-extractable value is even higher. there are all discussions that we need to have, and challenges that we need to address, if we want the ethereum community to be a leader in implementing decentralized, credibly neutral and market-based solutions to public goods funding challenges. dark mode toggle how do trusted setups work? 2022 mar 14 see all posts necessary background: elliptic curves and elliptic curve pairings. see also: dankrad feist's article on kzg polynomial commitments. special thanks to justin drake, dankrad feist and chih-cheng liang for feedback and review. many cryptographic protocols, especially in the areas of data availability sampling and zk-snarks depend on trusted setups. a trusted setup ceremony is a procedure that is done once to generate a piece of data that must then be used every time some cryptographic protocol is run. generating this data requires some secret information; the "trust" comes from the fact that some person or some group of people has to generate these secrets, use them to generate the data, and then publish the data and forget the secrets. but once the data is generated, and the secrets are forgotten, no further participation from the creators of the ceremony is required. there are many types of trusted setups. the earliest instance of a trusted setup being used in a major protocol is the original zcash ceremony in 2016. this ceremony was very complex, and required many rounds of communication, so it could only have six participants. everyone using zcash at that point was effectively trusting that at least one of the six participants was honest. more modern protocols usually use the powers-of-tau setup, which has a 1-of-n trust model with \(n\) typically in the hundreds. that is to say, hundreds of people participate in generating the data together, and only one of them needs to be honest and not publish their secret for the final output to be secure. well-executed setups like this are often considered "close enough to trustless" in practice. this article will explain how the kzg setup works, why it works, and the future of trusted setup protocols. anyone proficient in code should also feel free to follow along this code implementation: https://github.com/ethereum/research/blob/master/trusted_setup/trusted_setup.py. what does a powers-of-tau setup look like? a powers-of-tau setup is made up of two series of elliptic curve points that look as follows: \([g_1, g_1 * s, g_1 * s^2 ... g_1 * s^{n_1-1}]\)   \([g_2, g_2 * s, g_2 * s^2 ... g_2 * s^{n_2-1}]\) \(g_1\) and \(g_2\) are the standardized generator points of the two elliptic curve groups; in bls12-381, \(g_1\) points are (in compressed form) 48 bytes long and \(g_2\) points are 96 bytes long. \(n_1\) and \(n_2\) are the lengths of the \(g_1\) and \(g_2\) sides of the setup. some protocols require \(n_2 = 2\), others require \(n_1\) and \(n_2\) to both be large, and some are in the middle (eg. ethereum's data availability sampling in its current form requires \(n_1 = 4096\) and \(n_2 = 16\)). \(s\) is the secret that is used to generate the points, and needs to be forgotten. to make a kzg commitment to a polynomial \(p(x) = \sum_i c_i x^i\), we simply take a linear combination \(\sum_i c_i s_i\), where \(s_i = g_1 * s^i\) (the elliptic curve points in the trusted setup). the \(g_2\) points in the setup are used to verify evaluations of polynomials that we make commitments to; i won't go into verification here in more detail, though dankrad does in his post. intuitively, what value is the trusted setup providing? it's worth understanding what is philosophically going on here, and why the trusted setup is providing value. a polynomial commitment is committing to a piece of size-\(n\) data with a size \(o(1)\) object (a single elliptic curve point). we could do this with a plain pedersen commitment: just set the \(s_i\) values to be \(n\) random elliptic curve points that have no known relationship with each other, and commit to polynomials with \(\sum_i c_i s_i\) as before. and in fact, this is exactly what ipa evaluation proofs do. however, any ipa-based proofs take \(o(n)\) time to verify, and there's an unavoidable reason why: a commitment to a polynomial \(p(x)\) using the base points \([s_0, s_1 ... s_i ... s_{n-1}]\) would commit to a different polynomial if we use the base points \([s_0, s_1 ... (s_i * 2) ... s_{n-1}]\). a valid commitment to the polynomial \(3x^3 + 8x^2 + 2x + 6\) under one set of base points is a valid commitment to \(3x^3 + 4x^2 + 2x + 6\) under a different set of base points. if we want to make an ipa-based proof for some statement (say, that this polynomial evaluated at \(x = 10\) equals \(3826\)), the proof should pass with the first set of base points and fail with the second. hence, whatever the proof verification procedure is cannot avoid somehow taking into account each and every one of the \(s_i\) values, and so it unavoidably takes \(o(n)\) time. but with a trusted setup, there is a hidden mathematical relationship between the points. it's guaranteed that \(s_{i+1} = s * s_i\) with the same factor \(s\) between any two adjacent points. if \([s_0, s_1 ... s_i ... s_{n-1}]\) is a valid setup, the "edited setup" \([s_0, s_1 ... (s_i * 2) ... s_{n-1}]\) cannot also be a valid setup. hence, we don't need \(o(n)\) computation; instead, we take advantage of this mathematical relationship to verify anything we need to verify in constant time. however, the mathematical relationship has to remain secret: if \(s\) is known, then anyone could come up with a commitment that stands for many different polynomials: if \(c\) commits to \(p(x)\), it also commits to \(\frac{p(x) * x}{s}\), or \(p(x) x + s\), or many other things. this would completely break all applications of polynomial commitments. hence, while some secret \(s\) must have existed at one point to make possible the mathematical link between the \(s_i\) values that enables efficient verification, the \(s\) must also have been forgotten. how do multi-participant setups work? it's easy to see how one participant can generate a setup: just pick a random \(s\), and generate the elliptic curve points using that \(s\). but a single-participant trusted setup is insecure: you have to trust one specific person! the solution to this is multi-participant trusted setups, where by "multi" we mean a lot of participants: over 100 is normal, and for smaller setups it's possible to get over 1000. here is how a multi-participant powers-of-tau setup works. take an existing setup (note that you don't know \(s\), you just know the points): \([g_1, g_1 * s, g_1 * s^2 ... g_1 * s^{n_1-1}]\)   \([g_2, g_2 * s, g_2 * s^2 ... g_2 * s^{n_2-1}]\) now, choose your own random secret \(t\). compute: \([g_1, (g_1 * s) * t, (g_1 * s^2) * t^2 ... (g_1 * s^{n_1-1}) * t^{n_2-1}]\)   \([g_2, (g_2 * s) * t, (g_2 * s^2) * t^2 ... (g_2 * s^{n_2-1}) * t^{n_2-1}]\) notice that this is equivalent to: \([g_1, g_1 * (st), g_1 * (st)^2 ... g_1 * (st)^{n_1-1}]\)   \([g_2, g_2 * (st), g_2 * (st)^2 ... g_2 * (st)^{n_2-1}]\) that is to say, you've created a valid setup with the secret \(s * t\)! you never give your \(t\) to the previous participants, and the previous participants never give you their secrets that went into \(s\). and as long as any one of the participants is honest and does not reveal their part of the secret, the combined secret does not get revealed. in particular, finite fields have the property that if you know know \(s\) but not \(t\), and \(t\) is securely randomly generated, then you know nothing about \(s*t\)! verifying the setup to verify that each participant actually participated, each participant can provide a proof that consists of (i) the \(g_1 * s\) point that they received and (ii) \(g_2 * t\), where \(t\) is the secret that they introduce. the list of these proofs can be used to verify that the final setup combines together all the secrets (as opposed to, say, the last participant just forgetting the previous values and outputting a setup with just their own secret, which they keep so they can cheat in any protocols that use the setup). \(s_1\) is the first participant's secret, \(s_2\) is the second participant's secret, etc. the pairing check at each step proves that the setup at each step actually came from a combination of the setup at the previous step and a new secret known by the participant at that step. each participant should reveal their proof on some publicly verifiable medium (eg. personal website, transaction from their .eth address, twitter). note that this mechanism does not prevent someone from claiming to have participated at some index where someone else has (assuming that other person has revealed their proof), but it's generally considered that this does not matter: if someone is willing to lie about having participated, they would also be willing to lie about having deleted their secret. as long as at least one of the people who publicly claim to have participated is honest, the setup is secure. in addition to the above check, we also want to verify that all the powers in the setup are correctly constructed (ie. they're powers of the same secret). to do this, we could do a series of pairing checks, verifying that \(e(s_{i+1}, g_2) = e(s_i, t_1)\) (where \(t_1\) is the \(g_2 * s\) value in the setup) for every \(i\). this verifies that the factor between each \(s_i\) and \(s_{i+1}\) is the same as the factor between \(t_1\) and \(g_2\). we can then do the same on the \(g_2\) side. but that's a lot of pairings and is expensive. instead, we take a random linear combination \(l_1 = \sum_{i=0}^{n_1-2} r_is_i\), and the same linear combination shifted by one: \(l_2 = \sum_{i=0}^{n_1-2} r_is_{i+1}\). we use a single pairing check to verify that they match up: \(e(l_2, g_2) = e(l_1, t_1)\). we can even combine the process for the \(g_1\) side and the \(g_2\) side together: in addition to computing \(l_1\) and \(l_2\) as above, we also compute \(l_3 = \sum_{i=0}^{n_2-2} q_it_i\) (\(q_i\) is another set of random coefficients) and \(l_4 = \sum_{i=0}^{n_2-2} q_it_{i+1}\), and check \(e(l_2, l_3) = e(l_1, l_4)\). setups in lagrange form in many use cases, you don't want to work with polynomials in coefficient form (eg. \(p(x) = 3x^3 + 8x^2 + 2x + 6\)), you want to work with polynomials in evaluation form (eg. \(p(x)\) is the polynomial that evaluates to \([19, 146, 9, 187]\) on the domain \([1, 189, 336, 148]\) modulo 337). evaluation form has many advantages (eg. you can multiply and sometimes divide polynomials in \(o(n)\) time) and you can even use it to evaluate in \(o(n)\) time. in particular, data availability sampling expects the blobs to be in evaluation form. to work with these cases, it's often convenient to convert the trusted setup to evaluation form. this would allow you to take the evaluations (\([19, 146, 9, 187]\) in the above example) and use them to compute the commitment directly. this is done most easily with a fast fourier transform (fft), but passing the curve points as input instead of numbers. i'll avoid repeating a full detailed explanation of ffts here, but here is an implementation; it is actually not that difficult. the future of trusted setups powers-of-tau is not the only kind of trusted setup out there. some other notable (actual or potential) trusted setups include: the more complicated setups in older zk-snark protocols (eg. see here), which are sometimes still used (particularly groth16) because verification is cheaper than plonk. some cryptographic protocols (eg. dark) depend on hidden-order groups, groups where it is not known what number an element can be multiplied by to get the zero element. fully trustless versions of this exist (see: class groups), but by far the most efficient version uses rsa groups (powers of \(x\) mod \(n = pq\) where \(p\) and \(q\) are not known). trusted setup ceremonies for this with 1-of-n trust assumptions are possible, but are very complicated to implement. if/when indistinguishability obfuscation becomes viable, many protocols that depend on it will involve someone creating and publishing an obfuscated program that does something with a hidden internal secret. this is a trusted setup: the creator(s) would need to possess the secret to create the program, and would need to delete it afterwards. cryptography continues to be a rapidly evolving field, and how important trusted setups are could easily change. it's possible that techniques for working with ipas and halo-style ideas will improve to the point where kzg becomes outdated and unnecessary, or that quantum computers will make anything based on elliptic curves non-viable ten years from now and we'll be stuck working with trusted-setup-free hash-based protocols. it's also possible that what we can do with kzg will improve even faster, or that a new area of cryptography will emerge that depends on a different kind of trusted setup. to the extent that trusted setup ceremonies are necessary, it is important to remember that not all trusted setups are created equal. 176 participants is better than 6, and 2000 would be even better. a ceremony small enough that it can be run inside a browser or phone application (eg. the zkopru setup is web-based) could attract far more participants than one that requires running a complicated software package. every ceremony should ideally have participants running multiple independently built software implementations and running different operating systems and environments, to reduce common mode failure risks. ceremonies that require only one round of interaction per participant (like powers-of-tau) are far better than multi-round ceremonies, both due to the ability to support far more participants and due to the greater ease of writing multiple implementations. ceremonies should ideally be universal (the output of one ceremony being able to support a wide range of protocols). these are all things that we can and should keep working on, to ensure that trusted setups can be as secure and as trusted as possible. getting to the frontier | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search getting to the frontier posted by vinay gupta on march 12, 2015 research & development so first some background. we’ve already covered a little about what to expect from frontier in the ethereum launch process blog post and gav has given us a comprehensive overview of the state of the development effort. you can also read about some of the ongoing security work, including the bug bounty system. the bug bounty program has found my favorite bug so far: jonas nick's discovery that you could send a negative payment to somebody which moves value from their account to yours! as bug bounty finds go, an absolute gem. well done! gav has a new blog post describing a little more about what the thorny end of the development process looks like: creating a precise specification for ethereum by keeping three different implementations in sync, rather than having a canonical implementation which implicitly defines the protocol spec and which other versions then become bug-for-bug compatible with. this is a pretty tough-minded and thorough approach to putting together a system, and it’s worth reading to get a flavor of how the engineering work is going. by now you should have a pretty good sense of what the development process looks like, and the disparate threads of work which are being woven together into the release. so let’s talk about what actually has to get done between now and frontier. there are four specific challenges for us to meet on frontier. security network stability mining exchange two of those, mining and exchange are partnership efforts: people have to take our software and run it, in concert with their own tooling in some cases, to provide services as part of the ecosystem. remember, we’re launching an ecosystem, not just a product: everything is part of that ecology. each one of these aspects of the release process has to work, both on its own, and with the others for the ecosystem to actually get a proper test under load. in addition: all software has to be reasonably easy to install, mining included we need to support partners, particularly exchanges, to provide services we need to build our checkpointing and blockchain integrity testing services all within the context of the frontier environment in which we make strong guarantees about network integrity (more on this later) at the cost of the risk of network rollbacks if/when we discover unexpected network behaviors or other problems. reasonably good progress is being made on all fronts. rather than giving you a release date, and then risking that under pressure to meet that date, something will slip through the gaps that another day of work would have caught, we are going to tick off items in the release process as we go. this will let you get have a sense of how far out from the launch we are on any given day, without us basically making a guess at how long it will all take, publishing our guess, and then hoping we can make reality to the calendar: as we said before, this is a security-driven not a schedule-driven process. we can make that process fairly transparent to you, and provide summary information so you don’t have to watch the repositories but we would rather bring you into the process so you understand what is happening and how the work is going than give you an aggregate date and keep you guessing as to the actual state of the work. we feel this approach is more transparent, better suited to a crowdfunded development process, and overall simply more realistic. if you would like to see the current state of play, jeffrey wilcke (lead on the go client and the amsterdam hub) has a go client issue tracker for frontier. this issue list is the best reference for the frontier release: right now, 10 issues closed, 26 open. new issues will get raised, and of course issue count is only a very loose proxy for “ship time” but the work done is the best indicator, so look there if you would like to prognosticate. please note: this is not to say that we are running on “valve time” (even though valve time does produce some spectacular products!) we have open development processes, you can see the code being written, download the latest branches, participate in testing. it’s not all hidden behind a company pr wall. but we are running on “releasing innovative software with huge associated security processes” time, and schedule estimation for things which have never been done before is not going to help anybody: part of the benefit of the way ethereum was funded over conventional funding models is that we can do what is right in the mediumand long-term, rather than being under constant market pressure for next quarter’s numbers. we will say this: there will be at least two weeks notice before anything goes live. you’ll get plenty of news from blog posts, you’ll see the issue count go down, and we’ll start saying positive things about security audit feedback and so on. we will keep you in the loop! what about the feature list? we’ve largely gone over this in the launch processes post, but to reiterate frontier will... be command-line client only mine at 10% of the normal rate, but this will be real ether include the full blockchain feature set including smart contracts and logs, although everything except account balances will be erased when homestead is launched we are working with exchanges to make eth convertible during frontier for security, the block chain will be checkpointed manually every 24 hours, and any reported anomalies will be investigated official exchanges will use this checkpointing service to protect traded assets from potential blockchain rollbacks in particularly severe cases of failure, the foundation may stop checkpointing frontier completely and release a client upgrade in short, we’re doing everything possible to make frontier a safe place to test ethereum with real value, but we are also strongly discouraging people from using ether on the frontier network which they are unwilling to lose. frontier is intended largely for people who are writing and testing tooling for mining and exchanges, and perhaps a few of the more hardy dapp developers. it is not a general release that we expect ordinary users to interact with at all, although you might download a client and mine a little ether just because you can. the show really starts at homestead, and we’ll have more news about homestead’s features later more news as i have it, and keep watching the skies! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum community survey | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum community survey posted by george hallam on january 9, 2015 research & development back in november, we created a quick survey for the ethereum community to help us gauge how we're doing, what can be improved, and how best we can engage with you all as we move forward towards the genesis block release in march. we feel it's very important to enable the community to interact with ethereum as well as itself, and we hope to offer new and exciting tools to do so using the survey results for guidance. the survey itself consisted of 14 questions split into two sections; ethereum as an "organisation" and ethereum as a "technology".  there was a total of 286 responses. this represents 7.8% of the current ethereum reddit population, or 2.4% of the current @ethereumproject followers. what country do you currently reside in? so, this is where everybody lives. to sum it up by continent of the 286 respondents there are 123 (43%) in north america, 114 (40%) in europe, 30 (10%) in asia, 13 (5%) in oceana and 6 (2%) in south america. no surprises there, though it does show how we and the crypto space in general have much work to do in areas south of the brandt line. one way to go about this is to seed more international ethereum meetups. you can see a map of all the current ethereum meetups here (we have 81 in total all over the world from london to new york to tehran with over 6000 members taking part). if you'd like to start one yourself, please do message us and we can offer further assistance - info@ethereum.org. it's understood that our transparency is very important to the community. to that end, we strive to make much of our internal workings freely available on the internet. as indicated in the chart, most people agree that we are doing just that. however, more can always be done. we're currently working on a refresh of the ethereum.org website ready for the release of the genesis block. expect much more content and information as we complete this towards the end of january. in the meantime, have a look at the ethereum github repository, or head over to the new ξth ðξv website for a greater understanding of the entity that is delivering ethereum 1.0, as well as its truly incredible team.   we've always tried to give the community as much information about our financial situation as possible, and from the results it seems like a lot of you agree. for further information on how ethereum intends to use the revenue generated in the ether sale as we move forward, check out the road map and the đξv plan. to learn more about the ether sale itself, have a look at vitalik's ether sale introduction, the ethereum bitcoin wallet, or the ether sale statistical overview. though most people agree ethereum's use cases in society are clear, i wouldn't be so sure we've figured them all out just yet. everyday we're speaking with developers and entrepreneurs via skype or on irc (join in your browser #ethereum / #ethereum-dev) who have thought of new and exciting ideas that they are looking to implement on top of ethereum many of which are brand new to us. for a brief overview of some of the use cases we've encountered, check out stephan tual's recent presentation at newfinance. we're doing our best to keep everyone updated with the plethora of changes, updates and general progression of the project that's been taking place over the recent months. gavin wood and jeff wilcke especially have written some excellent blog updates on how things are going in their respective berlin and amsterdam ðξv hubs. ξth ðξv's mission statement is now proudly presented on the ξth ðξv website for all to see. in detail, it explains what needs to be achieved as time goes on, but can be summed up as "to research, design and build software that, as best as possible, facilitates, in a secure, decentralised and fair manner, the communication and automatically-enforced agreement between parties." much like the crypto space in general, ethereum is somewhat difficult to initially get your head around. no doubt about that, and it's our job to make the process of gaining understanding and enabling participation as easy and intuitive as possible. as mentioned previously, the new look ethereum.org website will be an invaluable tool in helping people access the right information that is applicable to their own knowledge and skill set. also, in time we aim to create a udemy/codacademy like utility which will allow people with skills ranging from none to jedi master to learn how ethereum works and how to implement their ideas. in the mean time, a great place to start for those wanting to use ethereum is ken kappler's recent tutorials. this was an important question as it gave a lot of perspective on what aspects needed to be focused on before genesis, and what (though useful) could be developed afterwards. from a ui point of view, the go team in amsterdam is working towards the creation of mist, ethereum's "ðapp navigator". mist's initial design ideas are presented by the lead ui designer, alex van de sande in this video. ease of installation will factor greatly in user adoption we cant very well have people recompiling the client every time a new update is pushed! so binaries with internal update systems are in the pipeline. client reliability (bugs) is being actioned on by jutta steiner, the manager of our internal and external security audits. we expect the community bug bounty project to be live by the middle of january, so stay tuned and be ready for epic 11 figure satoshi rewards, leaderboards and more "1337" prizes. developer tools are on the way too. specifically, project "mix". mix supports some rather amazing features, including documentation, a compiler, debugger integration for writing information on code health, valid invariant, code structure and code formatting, as well as variable values and assertion truth annotations. it's a long term project expected to be delivered in the next 12-18 months, right now we are very much focused on completing the blockchain. once complete, we can reallocate our resources to other important projects. you can find out more in the mix presentation from ðξvcon-0. for now, documentation is constantly being generated on the ethereum github wiki. the blog and social media interaction will continue to deliver ethereum content on relevant channels with the aim of reaching the widest range of people as possible.   with more people owning smartphones than computers already, imagine how prolific they'll will be as time goes on? this will be the case especially in emerging markets such as india and nigeria, it's likely they'll leapfrog computers to some extent and gain wide adoption very quickly.  a mobile light client will be greatly important to the usability of ethereum. as part of ibm and samsung's joint project "adept" (an iot platform which is currently being unveiled at ces 2015), an android version of the ethereum java client ethereumj, is going to be open-sourced on github. this will go a long way to getting ethereum mobile! it's interesting to see a very mixed bag of responses for this question. as was said previously, ethereum's use cases are as wide as they are varied, and it's great to see how many different types of services people are looking to implement on top of ethereum. the emphasis on governance based ðapps highlights ethereum's ability to facilitate interactions between the digital and physical world and create autonomously governed communities that can compete with both governments and corporations. primavera de filippi and raffaele mauro investigate this further in the internet policy review journal. this chart shows a reasonably even spread, we've done our best to make the various clients available on different operating systems. you can find the alethzero binaries here, and the mist binaries here. these however become obsolete very quickly and may not connect to the test net as development continues, so if you considering using ethereum before release, it's well worth while checking the client building tutorials to get the most up to date versions of the clients.   with mist (go), alethzero (c++), pythereum (python) node-ethereum (node.js), and ethereumj (java), ethereum already has a plethora of clients available. the yellow paper written by gavin wood is a great reference for the community to create its own clients, as seen with those still under development such as the clojure and objective c iterations. as gavin wood has mentioned in a previous blogpost, mutan and lll as smart contract languages will be mothballed. serpent will be continued to be developed by vitalik with his team, and soldity will continue as the primary development language for ethereum contracts. you can try solidity in your browser, or watch the recent vision and roadmap presentation by gavin wood and vitalik buterin at ðξvcon-0. thanks to alex van de sande for helping with the implementation of the survey and chart graphics. icons retrieved from icons8. if anyone would like a copy of the raw survey results, feel free to email george@ethdev.com. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle make ethereum cypherpunk again 2023 dec 28 see all posts special thanks to paul dylan-ennis for feedback and review. one of my favorite memories from ten years ago was taking a pilgrimage to a part of berlin that was called the bitcoin kiez: a region in kreuzberg where there were around a dozen shops within a few hundred meters of each other that were all accepting bitcoin for payments. the centerpiece of this community was room 77, a restaurant and bar run by joerg platzer. in addition to simply accepting bitcoin, it also served as a community center, and all kinds of open source developers, political activists of various affiliations, and other characters would frequently come by. room 77, 2013. source: my article from 2013 on bitcoin magazine. a similar memory from two months earlier was porcfest (that's "porc" as in "porcupine" as in "don't tread on me"), a libertarian gathering in the forests of northern new hampshire, where the main way to get food was from small popup restaurants with names like "revolution coffee" and "seditious soups, salads and smoothies", which of course accepted bitcoin. here too, discussing the deeper political meaning of bitcoin, and using it in daily life, happened together side by side. the reason why i bring these memories up is that they remind me of a deeper vision underlying crypto: we are not here to just create isolated tools and games, but rather build holistically toward a more free and open society and economy, where the different parts technological, social and economic fit into each other. the early vision of "web3" was also a vision of this type, going in a similarly idealistic but somewhat different direction. the term "web3" was originally coined by ethereum cofounder gavin wood, and it refers to a different way of thinking about what ethereum is: rather than seeing it, as i initially did, as "bitcoin plus smart contracts", gavin thought about it more broadly as one of a set of technologies that could together form the base layer of a more open internet stack. a diagram that gavin wood used in many of his early presentations. when the free open source software movement began in the 1980s and 1990s, the software was simple: it ran on your computer and read and wrote to files that stayed on your computer. but today, most of our important work is collaborative, often on a large scale. and so today, even if the underlying code of an application is open and free, your data gets routed through a centralized server run by a corporation that could arbitrarily read your data, change the rules on you or deplatform you at any time. and so if we want to extend the spirit of open source software to the world of today, we need programs to have access to a shared hard drive to store things that multiple people need to modify and access. and what is ethereum, together with sister technologies like peer-to-peer messaging (then whisper, now waku) and decentralized file storage (then just swarm, now also ipfs)? a public decentralized shared hard drive. this is the original vision from which the now-ubiquitous term "web3" was born. unfortunately, since 2017 or so, these visions have faded somewhat into the background. few talk about consumer crypto payments, the only non-financial application that is actually being used at a large scale on-chain is ens, and there is a large ideological rift where significant parts of the non-blockchain decentralization community see the crypto world as a distraction, and not as a kindred spirit and a powerful ally. in many countries, people do use cryptocurrency to send and save money, but they often do this through centralized means: either through internal transfers on centralized exchange accounts, or by trading usdt on tron. background: the humble tron founder and decentralization pioneer justin sun bravely leading forth the coolest and most decentralized crypto ecosystem in the global world. having lived through that era, the number one culprit that i would blame as the root cause of this shift is the rise in transaction fees. when the cost of writing to the chain is $0.001, or even $0.1, you could imagine people making all kinds of applications that use blockchains in various ways, including non-financial ways. but when transaction fees go to over $100, as they have during the peak of the bull markets, there is exactly one audience that remains willing to play and in fact, because coin prices are going up and they're getting richer, becomes even more willing to play: degen gamblers. degen gamblers can be okay in moderate doses, and i have talked to plenty of people at events who were motivated to join crypto for the money but stayed for the ideals. but when they are the largest group using the chain on a large scale, this adjusts the public perception and the crypto space's internal culture, and leads to many of the other negatives that we have seen play out over the last few years. now, fast forward to 2023. on both the core challenge of scaling, and on various "side quests" of crucial importance to building a cypherpunk future actually viable, we actually have a lot of positive news to show: rollups are starting to actually exist. following a temporary lull after the regulatory crackdowns on tornado cash, second-generation privacy solutions such as railway and nocturne are seeing the (moon) light. account abstraction is starting to take off. light clients, forgotten for a long time, are starting to actually exist. zero knowledge proofs, a technology which we thought was decades away, are now here, are increasingly developer-friendly, and are on the cusp of being usable for consumer applications. these two things: the growing awareness that unchecked centralization and over-financialization cannot be what "crypto is about", and the key technologies mentioned above that are finally coming to fruition, together present us with an opportunity to take things in a different direction. namely, to make at least a part of the ethereum ecosystem actually be the permissionless, decentralized, censorship resistant, open source ecosystem that we originally came to build. what are some of these values? many of these values are shared not just by many in the ethereum community, but also by other blockchain communities, and even non-blockchain decentralization communities, though each community has its own unique combination of these values and how much each one is emphasized. open global participation: anyone in the world should be able to participate as a user, observer or developer, on a maximally equal footing. participation should be permissionless. decentralization: minimize the dependence of an application on any one single actor. in particular, an application should continue working even if its core developers disappear forever. censorship resistance: centralized actors should not have the power to interfere with any given user's or application's ability to operate. concerns around bad actors should be addressed at higher layers of the stack. auditability: anyone should be able to validate an application's logic and its ongoing operation (eg. by running a full node) to make sure that it is operating according to the rules that its developers claim it is. credible neutrality: base-layer infrastructure should be neutral, and in such a way that anyone can see that it is neutral even if they do not already trust the developers. building tools, not empires. empires try to capture and trap the user inside a walled garden; tools do their task but otherwise interoperate with a wider open ecosystem. cooperative mindset: even while competing, projects within the ecosystem cooperate on shared software libraries, research, security, community building and other areas that are commonly valuable to them. projects try to be positive-sum, both with each other and with the wider world. it is very possible to build things within the crypto ecosystem that do not follow these values. one can build a system that one calls a "layer 2", but which is actually a highly centralized system secured by a multisig, with no plans to ever switch to something more secure. one can build an account abstraction system that tries to be "simpler" than erc-4337, but at the cost of introducing trust assumptions that end up removing the possibility of a public mempool and make it much harder for new builders to join. one could build an nft ecosystem where the contents of the nft are needlessly stored on centralized websites, making it needlessly more fragile than if those compoents are stored on ipfs. one could build a staking interface that needlessly funnels users toward the already-largest staking pool. resisting these pressures is hard, but if we do not do so, then we risk losing the unique value of the crypto ecosystem, and recreating a clone of the existing web2 ecosystem with extra inefficiencies and extra steps. it takes a sewer to make a ninja turtle the crypto space is in many ways an unforgiving environment. a 2021 article by dan robinson and georgios konstantiopoulos expresses this vividly in the context of mev, arguing that ethereum is a dark forest where on-chain traders are constantly vulnerable to getting exploited by front-running bots, those bots themselves are vulnerable to getting counter-exploited by other bots, etc. this is also true in other ways: smart contracts regularly get hacked, users' wallets regularly get hacked, centralized exchanges fail even more spectacularly, etc. this is a big challenge for users of the space, but it also presents an opportunity: it means that we have a space to actually experiment with, incubate and receive rapid live feedback on all kinds of security technologies to address these challenges. we have seen successful responses to challenges in various contexts already: problem solution centralized exchages getting hacked use dexes plus stablecoins, so centralized entities only need to be trusted to handle fiat individual private keys are not secure smart contract wallets: multisig, social recovery, etc users getting tricked into signing transactions that drain their money wallets like rabby showing their users results of transaction simulation users getting sandwich-attacked by mev players cowswap, flashbots protect, mev blocker... everyone wants the internet to be safe. some attempt to make the internet safe by pushing approaches that force reliance on a single particular actor, whether a corporation or a government, that can act as a centralized anchor of safety and truth. but these approaches sacrifice openness and freedom, and contribute to the tragedy that is the growing "splinternet". people in the crypto space highly value openness and freedom. the level of risks and the high financial stakes involved mean that the crypto space cannot ignore safety, but various ideological and structural reasons ensure that centralized approaches for achieving safety are not available to it. at the same time, the crypto space is at the frontier of very powerful technologies like zero knowledge proofs, formal verification, hardware-based key security and on-chain social graphs. these facts together mean that, for crypto, the open way to improving security is the only way. all of this is to say, the crypto world is a perfect testbed environment to take its open and decentralized approach to security and actually apply it in a realistic high-stakes environment, and mature it to the point where parts of it can then be applied in the broader world. this is one of my visions for how the idealistic parts of the crypto world and the chaotic parts of the crypto world, and then the crypto world as a whole and the broader mainstream, can turn their differences into a symbiosis rather than a constant and ongoing tension. ethereum as part of a broader technological vision in 2014, gavin wood introduced ethereum as one of a suite of tools that can be built, the other two being whisper (decentralized messaging) and swarm (decentralized storage). the former was heavily emphasized, but with the turn toward financialization around 2017 the latter were unfortunately given much less love and attention. that said, whisper continues to exist as waku, and is being actively used by projects like the decentralized messenger status. swarm continues to be developed, and now we also have ipfs, which is used to host and serve this blog. in the last couple of years, with the rise of decentralized social media (lens, farcaster, etc), we have an opportunity to revisit some of these tools. in addition, we also have another very powerful new tool to add to the trifecta: zero knowledge proofs. these technologies are most widely adopted as ways of improving ethereum's scalability, as zk rollups, but they are also very useful for privacy. in particular, the programmability of zero knowlege proofs means that we can get past the false binary of "anonymous but risky" vs "kyc'd therefore safe", and get privacy and many kinds of authentication and verification at the same time. an example of this in 2023 was zupass. zupass is a zero-knowledge-proof-based system that was incubated at zuzalu, which was used both for in-person authentication to events, and for online authentication to the polling system zupoll, the twitter-lookalike zucast and others. the key feature of zupass was this: you can prove that you are a resident of zuzalu, without revealing which member of zuzalu you are. furthermore, each zuzalu resident could only have one randomly-generated cryptographic identity for each application instance (eg. a poll) that they were signing into. zupass was highly successful, and was applied later in the year to do ticketing at devconnect. a zero-knowledge proof proving that i, as an ethereum foundation employee, have access to the devconnect coworking space. the most practical use of zupass so far has probably been the polling. all kinds of polls have been made, some on politically controversial or highly personal topics where people feel a strong need to preserve their privacy, using zupass as an anonymous voting platform. here, we can start to see the contours of what an ethereum-y cypherpunk world would look like, at least on a pure technical level. we can be holding our assets in eth and erc20 tokens, as well as all kinds of nfts, and use privacy systems based on stealth addresses and privacy pools technology to preserve our privacy while at the same time locking out known bad actors' ability to benefit from the same anonymity set. whether within our daos, or to help decide on changes to the ethereum protocol, or for any other objective, we can use zero-knowledge voting systems, which can use all kinds of credentials to help identify who has standing to vote and who does not: in addition to voting-with-tokens as done in 2017, we can have anonymous polls of people who have made sufficient contributions to the ecosystem, people who have attended enough events, or one-vote-per-person. in-person and online payments can happen with ultra-cheap transactions on l2s, which take advantage of data availability space (or off-chain data secured with plasma) together with data compression to give their users ultra-high scalability. payments from one rollup to another can happen with decentralized protocols like uniswapx. decentralized social media projects can use various storage layers to store activity such as posts, retweets and likes, and use ens (cheap on l2 with ccip) for usernames. we can have seamless integration between on-chain tokens, and off-chain attestations held personally and zk-proven through systems like zupass. mechanisms like quadratic voting, cross-tribal consensus finding and prediction markets can be used to help organizations and communities govern themselves and stay informed, and blockchain and zk-proof-based identities can make these systems secure against both centralized censorship from the inside and coordinated manipulation from the outside. sophisticated wallets can protect people as they participate in dapps, and user interfaces can be published to ipfs and accessed as .eth domains, with hashes of the html, javascript and all software dependencies updated directly on-chain through a dao. smart contract wallets, born to help people not lose tens of millions of dollars of their cryptocurrency, would expand to guard people's "identity roots", creating a system that is even more secure than centralized identity providers like "sign in with google". soul wallet recovery interface. i personally am at the point of being more willing to trust my funds and identity to systems like this than to centralized web2 recovery already. we can think of the greater ethereum-verse (or "web3") as creating an independent tech protocol stack, that is competing with the traditional centralized protocol stack at all levels. many people will mix-and-match both, and there are often clever ways to match both: with zkemail, you can even make an email address be one of the guardians of your social recovery wallet! but there are also many synergies from using the different parts of the decentralized stack together, especially if they are designed to better integrate with each other. traditional stack decentralized stack banking system eth, stablecoins, l2s for payments, dexes (note: still need banks for loans) receipts links to transactions on block explorers corporations daos dns (.com, .io, etc) ens (.eth) regular email encrypted email (eg. skiff) regular messaging (eg. telegram) decentralized messaging (eg. status) sign in with google, twitter, wechat sign in with ethereum, zupass, attestations via eas, poaps, zu-stamps... + social recovery publishing blogs on medium, etc publishing self-hosted blogs on ipfs (eg. using fleek) twitter, facebook lens, farcaster... limit bad actors through all-seeing big brother constrain bad actors through zero knowledge proofs one of the benefits of thinking about it as a stack is that this fits well with ethereum's pluralist ethos. bitcoin is trying to solve one problem, or at most two or three. ethereum, on the other hand, has lots of sub-communities with lots of different focuses. there is no single dominant narrative. the goal of the stack is to enable this pluralism, but at the same time strive for growing interoperability across this plurality. the social layer it's easy to say "these people doing x are a corrupting influence and bad, these people doing y are the real deal". but this is a lazy response. to truly succeed, we need not only a vision for a technical stack, but also the social parts of the stack that make the technical stack possible to build in the first place. the advantage of the ethereum community, in principle, is that we take incentives seriously. pgp wanted to put cryptographic keys into everyone's hands so we can actually do signed and encrypted email for decades, it largely failed, but then we got cryptocurrency and suddenly millions of people have keys publicly associated to them, and we can start using those keys for other purposes including going full circle back to encrypted email and messaging. non-blockchain decentralization projects are often chronically underfunded, blockchain-based projects get a 50-million dollar series b round. it is not from the benevolence of the staker that we get people to put in their eth to protect the ethereum network, but rather from their regard to their own self-interest and we get $20 billion in economic security as a result. at the same time, incentives are not enough. defi projects often start humble, cooperative and maximally open source, but sometimes begin to abandon these ideals as they grow in size. we can incentivize stakers to come and participate with very high uptime, but is much more difficult to incentivize stakers to be decentralized. it may not be doable using purely in-protocol means at all. lots of critical pieces of the "decentralized stack" described above do not have viable business models. the ethereum protocol's governance itself is notably non-financialized and this has made it much more robust than other ecosystems whose governance is more financialized. this is why it's valuable for ethereum to have a strong social layer, which vigorously enforces its values in those places where pure incentives can't but without creating a notion of "ethereum alignment" that turns into a new form of political correctness. there is a balance between these two sides to be made, though the right term is not so much balance as it is integration. there are plenty of people whose first introduction to the crypto space is the desire to get rich, but who then get acquainted with the ecosystem and become avid believers in the quest to build a more open and decentralized world. how do we actually make this integration happen? this is the key question, and i suspect the answer lies not in one magic bullet, but in a collection of techniques that will be arrived at iteratively. the ethereum ecosystem is already more successful than most in encouraging a cooperative mentality between layer 2 projects purely through social means. large-scale public goods funding, especially gitcoin grants and optimism's retropgf rounds, is also extremely helpful, because it creates an alternative revenue channel for developers that don't see any conventional business models that do not require sacrificing on their values. but even these tools are still in their infancy, and there is a long way to go to both improve these particular tools, and to identify and grow other tools that might be a better fit for specific problems. this is where i see the unique value proposition of ethereum's social layer. there is a unique halfway-house mix of valuing incentives, but also not getting consumed by them. there is a unqiue mix of valuing a warm and cohesive community, but at the same time remembering that what feels "warm and cohesive" from the inside can easily feel "oppressive and exclusive" from the outside, and valuing hard norms of neutrality, open source and censorship resistance as a way of guarding against the risks of going too far in being community-driven. if this mix can be made to work well, it will in turn be in the best possible position to realize its vision on the economic and technical level. dark mode toggle gitcoin grants round 6 retrospective 2020 jul 22 see all posts round 6 of gitcoin grants has just finished, with $227,847 in contributions from 1,526 contributors and $175,000 in matched funds distributed across 695 projects. this time around, we had three categories: the two usual categories of "tech" and "community" (the latter renamed from "media" to reflect a desire for a broad emphasis), and the round-6-special category crypto for black lives. first of all, here are the results, starting with the tech and community sections: stability of income in the last round, one concern i raised was stability of income. people trying to earn a livelihood off of quadratic funding grants would want to have some guarantee that their income isn't going to completely disappear in the next round just because the hive mind suddenly gets excited about something else. round 6 had two mechanisms to try to provide more stability of income: a "shopping cart" interface for giving many contributions, with an explicit "repeat your contributions from the last round" feature a rule that the matching amounts are calculated using not just contributions from this round, but also "carrying over" 1/3 of the contributions from the previous round (ie. if you made a $10 grant in the previous round, the matching formula would pretend you made a $10 grant in the previous round and also a $3.33 grant this round) was clearly successful at one goal: increasing the total number of contributions. but its effect in ensuring stability of income is hard to measure. the effect of (2), on the other hand, is easy to measure, because we have stats for the actual matching amount as well as what the matching amount "would have been" if the 1/3 carry-over rule was not in place. first from the tech category: now from the community category: clearly, the rule helps reduce volatility, pretty much exactly as expected. that said, one could argue that this result is trivial: you could argue that all that's going on here is something very similar to grabbing part of the revenue from round n (eg. see how the new eip-1559 community fund earned less than it otherwise would have) and moving it into round n+1. sure, numerically speaking the revenues are more "stable", but individual projects could have just provided this stability to themselves by only spending 2/3 of the pot from each round, and using the remaining third later when some future round is unexpectedly low. why should the quadratic funding mechanism significantly increase its complexity just to achieve a gain in stability that projects could simply provide for themselves? my instinct says that it would be best to try the next round with the "repeat last round" feature but without the 1/3 carryover, and see what happens. particularly, note that the numbers seem to show that the media section would have been "stable enough" even without the carryover. the tech section was more volatile, but only because of the sudden entrance of the eip 1559 community fund; it would be part of the experiment to see just how common that kind of situation is. about that eip 1559 community fund... the big unexpected winner of this round was the eip 1559 community fund. eip 1559 (eip here, faq here, original paper here) is a major fee market reform proposal which far-reaching consequences; it aims to improve the user experience of sending ethereum transactions, reduce economic inefficiencies, provide an accurate in-protocol gas price oracle and burn a portion of fee revenue. many people in the ethereum community are very excited about this proposal, though so far there has been fairly little funding toward getting it implemented. this gitcoin grant was a large community effort toward fixing this. the grant had quite a few very large contributions, including roughly $2,400 each from myself and eric conner, early on. early in the round, one could clearly see the eip 1559 community grant having an abnormally low ratio of matched funds to contributed funds; it was somewhere around $4k matched to $20k contributed. this was because while the amount contributed was large, it came from relatively few wealthier donors, and so the matching amount was less than it would have been had the same quantity of funds come from more diverse sources the quadratic funding formula working as intended. however, a social media push advertising the grant then led to a large number of smaller contributors following along, which then quickly raised the match to its currently very high value ($35,578). quadratic signaling unexpectedly, this grant proved to have a double function. first, it provided $65,473 of much-needed funding to eip 1559 implementation. second, it served as a credible community signal of the level of demand for the proposal. the ethereum community has long been struggling to find effective ways to determine what "the community" supports, especially in cases of controversy. coin votes have been used in the past, and have the advantage that they come with an answer to the key problem of determining who is a "real community member" the answer is, your membership in the ethereum community is proportional to how much eth you have. however, they are plutocratic; in the famous dao coin vote, a single "yes" voter voted with more eth than all "no" voters put together (~20% of the total). the alternative, looking at github, reddit and twitter comments and votes to measure sentiment (sometimes derided as "proof of social media") is egalitarian, but it is easily exploitable, comes with no skin-in-the-game, and frequently falls under criticisms of "foreign interference" (are those really ethereum community members disagreeing with the proposal, or just those dastardly bitcoiners coming in from across the pond to stir up trouble?). quadratic funding falls perfectly in the middle: the need to contribute monetary value to vote ensures that the votes of those who really care about the project count more than the votes of less-concerned outsiders, and the square-root function ensures that the votes of individual ultra-wealthy "whales" cannot beat out a poorer, but broader, coalition. a diagram from my post on quadratic payments showing how quadratic payments is "in the middle" between the extremes of voting-like systems and money-like systems, and avoids the worst flaws of both. this raises the question: might it make sense to try to use explicit quadratic voting (with the ability to vote "yes" or "no" to a proposal) as an additional signaling tool to determine community sentiment for ethereum protocol proposals? how well are "guest categories" working? since round 5, gitcoin grants has had three categories per round: tech, community (called "media" before), and some "guest" category that appears only during that specific round. in round 5 this was covid relief; in round 6, it's crypto for black lives. by far the largest recipient was black girls code, claiming over 80% of the matching pot. my guess for why this happened is simple: black girls code is an established project that has been participating in the grants for several rounds already, whereas the other projects were new entrants that few people in the ethereum community knew well. in addition, of course, the ethereum community "understands" the value of helping people code more than it understands chambers of commerce and bail funds. this raises the question: is gitcoin's current approach of having a guest category each round actually working well? the case for "no" is basically this: while the individual causes (empowering black communities, and fighting covid) are certainly admirable, the ethereum community is by and large not experts at these topics, and we're certainly not experts on those specific projects working on those challenges. if the goal is to try to bring quadratic funding to causes beyond ethereum, the natural alternative is a separate funding round marketed specifically to those communities; https://downtownstimulus.com/ is a great example of this. if the goal is to get the ethereum community interested in other causes, then perhaps running more than one round on each cause would work better. for example, "guest categories" could last for three rounds (~6 months), with $8,333 matching per round (and there could be two or three guest categories running simultaneously). in any case, it seems like some revision of the model makes sense. collusion now, the bad news. this round saw an unprecedented amount of attempted collusion and other forms of fraud. here are a few of the most egregious examples. blatant attempted bribery: impersonation: many contributions with funds clearly coming from a single address: the big question is: how much fraudulent activity can be prevented in a fully automated/technological way, without requiring detailed analysis of each and every case? if quadratic funding cannot survive such fraud without needing to resort to expensive case-by-case judgement, then regardless of its virtues in an ideal world, in reality it would not be a very good mechanism! fortunately, there is a lot that we can do to reduce harmful collusion and fraud that we are not yet doing. stronger identity systems is one example; in this round, gitcoin added optional sms verification, and it seems like the in this round the detected instances of collusion were mostly github-verified accounts and not sms-verified accounts. in the next round, making some form of extra verification beyond a github account (whether sms or something more decentralized, eg. brightid) seems like a good idea. to limit bribery, maci can help, by making it impossible for a briber to tell who actually voted for any particular project. impersonation is not really a quadratic funding-specific challenge; this could be solved with manual verification, or if one wishes for a more decentralized solution one could try using kleros or some similar system. one could even imagine incentivized reporting: anyone can lay down a deposit and flag a project as fraudulent, triggering an investigation; if the project turns out to be legitimate the deposit is lost but if the project turns out to be fraudulent, the challenger gets half of the funds that were sent to that project. conclusion the best news is the unmentioned news: many of the positive behaviors coming out of the quadratic funding rounds have stabilized. we're seeing valuable projects get funded in the tech and community categories, there has been less social media contention this round than in previous rounds, and people are getting better and better at understanding the mechanism and how to participate in it. that said, the mechanism is definitely at a scale where we are seeing the kinds of attacks and challenges that we would realistically see in a larger-scale context. there are some challenges that we have not yet worked through (one that i am particularly watching out for is: matched grants going to a project that one part of the community supports and another part of the community thinks is very harmful). that said, we've gotten as far as we have with fewer problems than even i had been anticipating. i recommend holding steady, focusing on security (and scalability) for the next few rounds, and coming up with ways to increase the size of the matching pots. and i continue to look forward to seeing valuable public goods get funded! superrationality and daos | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search superrationality and daos posted by vitalik buterin on january 23, 2015 research & development warning: this post contains crazy ideas. myself describing a crazy idea should not be construed as implying that (i) i am certain that the idea is correct/viable, (ii) i have an even >50% probability estimate that the idea is correct/viable, or that (iii) "ethereum" endorses any of this in any way. one of the common questions that many in the crypto 2.0 space have about the concept of decentralized autonomous organizations is a simple one: what are daos good for? what fundamental advantage would an organization have from its management and operations being tied down to hard code on a public blockchain, that could not be had by going the more traditional route? what advantages do blockchain contracts offer over plain old shareholder agreements? particularly, even if public-good rationales in favor of transparent governance, and guarnateed-not-to-be-evil governance, can be raised, what is the incentive for an individual organization to voluntarily weaken itself by opening up its innermost source code, where its competitors can see every single action that it takes or even plans to take while themselves operating behind closed doors? there are many paths that one could take to answering this question. for the specific case of non-profit organizations that are already explicitly dedicating themselves to charitable causes, one can rightfully say that the lack of individual incentive; they are already dedicating themselves to improving the world for little or no monetary gain to themselves. for private companies, one can make the information-theoretic argument that a governance algorithm will work better if, all else being equal, everyone can participate and introduce their own information and intelligence into the calculation a rather reasonable hypothesis given the established result from machine learning that much larger performance gains can be made by increasing the data size than by tweaking the algorithm. in this article, however, we will take a different and more specific route. what is superrationality? in game theory and economics, it is a very widely understood result that there exist many classes of situations in which a set of individuals have the opportunity to act in one of two ways, either "cooperating" with or "defecting" against each other, such that everyone would be better off if everyone cooperated, but regardless of what others do each indvidual would be better off by themselves defecting. as a result, the story goes, everyone ends up defecting, and so people's individual rationality leads to the worst possible collective result. the most common example of this is the celebrated prisoner's dilemma game. since many readers have likely already seen the prisoner's dilemma, i will spice things up by giving eliezer yudkowsky's rather deranged version of the game: let's suppose that four billion human beings not the whole human species, but a significant part of it are currently progressing through a fatal disease that can only be cured by substance s. however, substance s can only be produced by working with [a strange ai from another dimension whose only goal is to maximize the quantity of paperclips] substance s can also be used to produce paperclips. the paperclip maximizer only cares about the number of paperclips in its own universe, not in ours, so we can't offer to produce or threaten to destroy paperclips here. we have never interacted with the paperclip maximizer before, and will never interact with it again. both humanity and the paperclip maximizer will get a single chance to seize some additional part of substance s for themselves, just before the dimensional nexus collapses; but the seizure process destroys some of substance s. the payoff matrix is as follows: humans cooperate humans defect ai cooperates 2 billion lives saved, 2 paperclips gained 3 billion lives, 0 paperclips ai defects 0 lives, 3 paperclips 1 billion lives, 1 paperclip from our point of view, it obviously makes sense from a practical, and in this case moral, standpoint that we should defect; there is no way that a paperclip in another universe can be worth a billion lives. from the ai's point of view, defecting always leads to one extra paperclip, and its code assigns a value to human life of exactly zero; hence, it will defect. however, the outcome that this leads to is clearly worse for both parties than if the humans and ai both cooperated but then, if the ai was going to cooperate, we could save even more lives by defecting ourselves, and likewise for the ai if we were to cooperate. in the real world, many two-party prisoner's dilemmas on the small scale are resolved through the mechanism of trade and the ability of a legal system to enforce contracts and laws; in this case, if there existed a god who has absolute power over both universes but cared only about compliance with one's prior agreements, the humans and the ai could sign a contract to cooperate and ask the god to simultaneously prevent both from defecting. when there is no ability to pre-contract, laws penalize unilateral defection. however, there are still many situations, particularly when many parties are involved, where opportunities for defection exist: alice is selling lemons in a market, but she knows that her current batch is low quality and once customers try to use them they will immediately have to throw them out. should she sell them anyway? (note that this is the sort of marketplace where there are so many sellers you can't really keep track of reputation). expected gain to alice: $5 revenue per lemon minus $1 shipping/store costs = $4. expected cost to society: $5 revenue minus $1 costs minus $5 wasted money from customer = -$1. alice sells the lemons. should bob donate $1000 to bitcoin development? expected gain to society: $10 * 100000 people $1000 = $999000, expected gain to bob: $10 $1000 = -$990, so bob does not donate. charlie found someone else's wallet, containing $500. should he return it? expected gain to society: $500 (to recipient) $500 (charlie's loss) + $50 (intangible gain to society from everyone being able to worry a little less about the safety of their wallets). expected gain to charlie: -$500, so he keeps the wallet. should david cut costs in his factory by dumping toxic waste into a river? expected gain to society: $1000 savings minus $10 average increased medical costs * 100000 people = -$999000, expected gain to david: $1000 $10 = $990, so david pollutes. eve developed a cure for a type of cancer which costs $500 per unit to produce. she can sell it for $1000, allowing 50,000 cancer patients to afford it, or for $10000, allowing 25,000 cancer patients to afford it. should she sell at the higher price? expected gain to society: -25,000 lives (including alice's profit, which cancels' out the wealthier buyers' losses). expected gain to eve: $237.5 million profit instead of $25 million = $212.5 million, so eve charges the higher price. of course, in many of these cases, people sometimes act morally and cooperate, even though it reduces their personal situation. but why do they do this? we were produced by evolution, which is generally a rather selfish optimizer. there are many explanations. one, and the one we will focus on, involves the concept of superrationality. superrationality consider the following explanation of virtue, courtesy of david friedman: i start with two observations about human beings. the first is that there is a substantial connection between what goes on inside and outside of their heads. facial expressions, body positions, and a variety of other signs give us at least some idea of our friends' thoughts and emotions. the second is that we have limited intellectual ability--we cannot, in the time available to make a decision, consider all options. we are, in the jargon of computers, machines of limited computing power operating in real time. suppose i wish people to believe that i have certain characteristics--that i am honest, kind, helpful to my friends. if i really do have those characteristics, projecting them is easy--i merely do and say what seems natural, without paying much attention to how i appear to outside observers. they will observe my words, my actions, my facial expressions, and draw reasonably accurate conclusions. suppose, however, that i do not have those characteristics. i am not (for example) honest. i usually act honestly because acting honestly is usually in my interest, but i am always willing to make an exception if i can gain by doing so. i must now, in many actual decisions, do a double calculation. first, i must decide how to act--whether, for example, this is a good opportunity to steal and not be caught. second, i must decide how i would be thinking and acting, what expressions would be going across my face, whether i would be feeling happy or sad, if i really were the person i am pretending to be. if you require a computer to do twice as many calculations, it slows down. so does a human. most of us are not very good liars. if this argument is correct, it implies that i may be better off in narrowly material terms--have, for instance, a higher income--if i am really honest (and kind and ...) than if i am only pretending to be, simply because real virtues are more convincing than pretend ones. it follows that, if i were a narrowly selfish individual, i might, for purely selfish reasons, want to make myself a better person--more virtuous in those ways that others value. the final stage in the argument is to observe that we can be made better--by ourselves, by our parents, perhaps even by our genes. people can and do try to train themselves into good habits--including the habits of automatically telling the truth, not stealing, and being kind to their friends. with enough training, such habits become tastes--doing "bad" things makes one uncomfortable, even if nobody is watching, so one does not do them. after a while, one does not even have to decide not to do them. you might describe the process as synthesizing a conscience. essentially, it is cognitively hard to convincingly fake being virtuous while being greedy whenever you can get away with it, and so it makes more sense for you to actually be virtuous. much ancient philosophy follows similar reasoning, seeing virtue as a cultivated habit; david friedman simply did us the customary service of an economist and converted the intuition into more easily analyzable formalisms. now, let us compress this formalism even further. in short, the key point here is that humans are leaky agents with every second of our action, we essentially indirectly expose parts of our source code. if we are actually planning to be nice, we act one way, and if we are only pretending to be nice while actually intending to strike as soon as our friends are vulnerable, we act differently, and others can often notice. this might seem like a disadvantage; however, it allows a kind of cooperation that was not possible with the simple game-theoretic agents described above. suppose that two agents, a and b, each have the ability to "read" whether or not the other is "virtuous" to some degree of accuracy, and are playing a symmetric prisoner's dilemma. in this case, the agents can adopt the following strategy, which we assume to be a virtuous strategy: try to determine if the other party is virtuous. if the other party is virtuous, cooperate. if the other party is not virtuous, defect. if two virtuous agents come into contact with each other, both will cooperate, and get a larger reward. if a virtuous agent comes into contact with a non-virtuous agent, the virtuous agent will defect. hence, in all cases, the virtuous agent does at least as well as the non-virtuous agent, and often better. this is the essence of superrationality. as contrived as this strategy seems, human cultures have some deeply ingrained mechanisms for implementing it, particularly relating to mistrusting agents who try hard to make themselves less readable see the common adage that you should never trust someone who doesn't drink. of course, there is a class of individuals who can convincingly pretend to be friendly while actually planning to defect at every moment these are called sociopaths, and they are perhaps the primary defect of this system when implemented by humans. centralized manual organizations... this kind of superrational cooperation has been arguably an important bedrock of human cooperation for the last ten thousand years, allowing people to be honest to each other even in those cases where simple market incentives might instead drive defection. however, perhaps one of the main unfortunate byproducts of the modern birth of large centralized organizations is that they allow people to effectively cheat others' ability to read their minds, making this kind of cooperation more difficult. most people in modern civilization have benefited quite handsomely, and have also indirectly financed, at least some instance of someone in some third world country dumping toxic waste into a river to build products more cheaply for them; however, we do not even realize that we are indirectly participating in such defection; corporations do the dirty work for us. the market is so powerful that it can arbitrage even our own morality, placing the most dirty and unsavory tasks in the hands of those individuals who are willing to absorb their conscience at lowest cost and effectively hiding it from everyone else. the corporations themselves are perfectly able to have a smiley face produced as their public image by their marketing departments, leaving it to a completely different department to sweet-talk potential customers. this second department may not even know that the department producing the product is any less virtuous and sweet than they are. the internet has often been hailed as a solution to many of these organizational and political problems, and indeed it does do a great job of reducing information asymmetries and offering transparency. however, as far as the decreasing viability of superrational cooperation goes, it can also sometimes make things even worse. online, we are much less "leaky" even as individuals, and so once again it is easier to appear virtuous while actually intending to cheat. this is part of the reason why scams online and in the cryptocurrency space are more common than offline, and is perhaps one of the primary arguments against moving all economic interaction to the internet a la cryptoanarchism (the other argument being that cryptoanarchism removes the ability to inflict unboundedly large punishments, weakening the strength of a large class of economic mechanisms). a much greater degree of transparency, arguably, offers a solution. individuals are moderately leaky, current centralized organizations are less leaky, but organizations where randomly information is constantly being released to the world left, right and center are even more leaky than individuals are. imagine a world where if you start even thinking about how you will cheat your friend, business partner or spouse, there is a 1% chance that the left part of your hippocampus will rebel and send a full recording of your thoughts to your intended victim in exchange for a $7500 reward. that is what it "feels" like to be the management board of a leaky organization. this is essentially a restatement of the founding ideology behind wikileaks, and more recently an incentivized wikileaks alternative, slur.io came out to push the envelope further. however, wikileaks exists, and yet shadowy centralized organizations also continue to still exist and are in many cases still quite shadowy. perhaps incentivization, coupled with prediction-like-mechanisms for people to profit from outing their employers' misdeeds, is what will open the floodgates for greater transparency, but at the same time we can also take a different route: offer a way for organizations to make themselves voluntarily, and radically, leaky and superrational to an extent never seen before. ... and daos decentralized autonomous organizations, as a concept, are unique in that their governance algorithms are not just leaky, but actually completely public. that is, while with even transparent centralized organizations outsiders can get a rough idea of what the organization's temperament is, with a dao outsiders can actually see the organization's entire source code. now, they do not see the "source code" of the humans that are behind the dao, but there are ways to write a dao's source code so that it is heavily biased toward a particular objective regardless of who its participants are. a futarchy maximizing the average human lifespan will act very differently from a futarchy maximizing the production of paperclips, even if the exact same people are running it. hence, not only is it the case that the organization will make it obvious to everyone if they start to cheat, but rather it's not even possible for the organization's "mind" to cheat. now, what would superrational cooperation using daos look like? first, we would need to see some daos actually appear. there are a few use-cases where it seems not too far-fetched to expect them to succeed: gambling, stablecoins, decentralized file storage, one-id-per-person data provision, schellingcoin, etc. however, we can call these daos type i daos: they have some internal state, but little autonomous governance. they cannot ever do anything but perhaps adjust a few of their own parameters to maximize some utility metric via pid controllers, simulated annealing or other simple optimization algorithms. hence, they are in a weak sense superrational, but they are also rather limited and stupid, and so they will often rely on being upgraded by an external process which is not superrational at all. in order to go further, we need type ii daos: daos with a governance algorithm capable of making theoretically arbitrary decisions. futarchy, various forms of democracy, and various forms of subjective extra-protocol governance (ie. in case of substantial disagreement, dao clones itself into multiple parts with one part for each proposed policy, and everyone chooses which version to interact with) are the only ones we are currently aware of, though other fundamental approaches and clever combinations of these will likely continue to appear. once daos can make arbitrary decisions, then they will be able to not only engage in superrational commerce with their human customers, but also potentially with each other. what kinds of market failures can superrational cooperation solve that plain old regular cooperation cannot? public goods problems may unfortunately be outside the scope; none of the mechanisms described here solve the massively-multiparty incentivization problem. in this model, the reason why organizations make themselves decentralized/leaky is so that others will trust them more, and so organizations that fail to do this will be excluded from the economic benefits of this "circle of trust". with public goods, the whole problem is that there is no way to exclude anyone from benefiting, so the strategy fails. however, anything related to information asymmetries falls squarely within the scope, and this scope is large indeed; as society becomes more and more complex, cheating will in many ways become progressively easier and easier to do and harder to police or even understand; the modern financial system is just one example. perhaps the true promise of daos, if there is any promise at all, is precisely to help with this. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle what do i think about community notes? 2023 aug 16 see all posts special thanks to dennis pourteaux and jay baxter for feedback and review. the last two years of twitter x have been tumultuous, to say the least. after the platform was bought not bought bought by elon musk for $44 billion last year, elon enacted sweeping changes to the company's staffing, content moderation and business model, not to mention changes to the culture on the site that may well have been a result of elon's soft power more than any specific policy decision. but in the middle of these highly contentious actions, one new feature on twitter grew rapidly in importance, and seems to be beloved by people across the political spectrum: community notes. community notes is a fact-checking tool that sometimes attaches context notes, like the one on elon's tweet above, to tweets as a fact-checking and anti-misinformation tool. it was originally called birdwatch, and was first rolled out as a pilot project in january 2021. since then, it has expanded in stages, with the most rapid phase of its expansion coinciding with twitter's takeover by elon last year. today, community notes appear frequently on tweets that get a very large audience on twitter, including those on contentious political topics. and both in my view, and in the view of many people across the political spectrum i talk to, the notes, when they appear, are informative and valuable. but what interests me most about community notes is how, despite not being a "crypto project", it might be the closest thing to an instantiation of "crypto values" that we have seen in the mainstream world. community notes are not written or curated by some centrally selected set of experts; rather, they can be written and voted on by anyone, and which notes are shown or not shown is decided entirely by an open source algorithm. the twitter site has a detailed and extensive guide describing how the algorithm works, and you can download the data containing which notes and votes have been published, run the algorithm locally, and verify that the output matches what is visible on the twitter site. it's not perfect, but it's surprisingly close to satisfying the ideal of credible neutrality, all while being impressively useful, even under contentious conditions, at the same time. how does the community notes algorithm work? anyone with a twitter account matching some criteria (basically: active for 6+ months, no recent rule violations, verified phone number) can sign up to participate in community notes. currently, participants are slowly and randomly being accepted, but eventually the plan is to let in anyone who fits the criteria. once you are accepted, you can at first participate in rating existing notes, and once you've made enough good ratings (measured by seeing which ratings match with the final outcome for that note), you can also write notes of your own. when you write a note, the note gets a score based on the reviews that it receives from other community notes members. these reviews can be thought of as being votes along a 3-point scale of helpful, somewhat_helpful and not_helpful, but a review can also contain some other tags that have roles in the algorithm. based on these reviews, a note gets a score. if the note's score is above 0.40, the note is shown; otherwise, the note is not shown. the way that the score is calculated is what makes the algorithm unique. unlike simpler algorithms, which aim to simply calculate some kind of sum or average over users' ratings and use that as the final result, the community notes rating algorithm explicitly attempts to prioritize notes that receive positive ratings from people across a diverse range of perspectives. that is, if people who usually disagree on how they rate notes end up agreeing on a particular note, that note is scored especially highly. let us get into the deep math of how this works. we have a set of users and a set of notes; we can create a matrix \(m\), where the cell \(m_{i,j}\) represents how the i'th user rated the j'th note. for any given note, most users have not rated that note, so most entries in the matrix will be zero, but that's fine. the goal of the algorithm is to create a four-column model of users and notes, assigning each user two stats that we can call "friendliness" and "polarity", and each note two stats that we can call "helpfulness" and "polarity". the model is trying to predict the matrix as a function of these values, using the following formula: note that here i am introducing both the terminology used in the birdwatch paper, and my own terms to provide a less mathematical intuition for what the variables mean: μ is a "general public mood" parameter that accounts for how high the ratings are that users give in general \(i_u\) is a user's "friendliness": how likely that particular user is to give high ratings \(i_n\) is a note's "helpfulness": how likely that particular note is to get rated highly. ultimately, this is the variable we care about. \(f_u\) or \(f_n\) is user or note's "polarity": its position among the dominant axis of political polarization. in practice, negative polarity roughly means "left-leaning" and positive polarity means "right-leaning", but note that the axis of polarization is discovered emergently from analyzing users and notes; the concepts of leftism and rightism are in no way hard-coded. the algorithm uses a pretty basic machine learning model (standard gradient descent) to find values for these variables that do the best possible job of predicting the matrix values. the helpfulness that a particular note is assigned is the note's final score. if a note's helpfulness is at least +0.4, the note gets shown. the core clever idea here is that the "polarity" terms absorb the properties of a note that cause it to be liked by some users and not others, and the "helpfulness" term only measures the properties that a note has that cause it to be liked by all. thus, selecting for helpfulness identifies notes that get cross-tribal approval, and selects against notes that get cheering from one tribe at the expense of disgust from the other tribe. i made a simplified implementation of the basic algorithm; you can find it here, and are welcome to play around with it. now, the above is only a description of the central core of the algorithm. in reality, there are a lot of extra mechanisms bolted on top. fortunately, they are described in the public documentation. these mechanisms include the following: the algorithm gets run many times, each time adding some randomly generated extreme "pseudo-votes" to the votes. this means that the algorithm's true output for each note is a range of values, and the final result depends on a "lower confidence bound" taken from this range, which is checked against a threshold of 0.32. if many users (especially users with a similar polarity to the note) rate a note "not helpful", and furthermore they specify the same "tag" (eg. "argumentative or biased language", "sources do not support note") as the reason for their rating, the helpfulness threshold required for the note to be published increases from 0.4 to 0.5 (this looks small but it's very significant in practice) if a note is accepted, the threshold that its helpfulness must drop below to de-accept it is 0.01 points lower than the threshold that a note's helpfulness needed to reach for the note to be originally accepted the algorithm gets run even more times with multiple models, and this can sometimes promote notes whose original helpfulness score is somewhere between 0.3 and 0.4 all in all, you get some pretty complicated python code that amounts to 6282 lines stretching across 22 files. but it is all open, you can download the note and rating data and run it yourself, and see if the outputs correspond to what is actually on twitter at any given moment. so how does this look in practice? probably the single most important idea in this algorithm that distinguishes it from naively taking an average score from people's votes is what i call the "polarity" values. the algorithm documentation calls them \(f_u\) and \(f_n\), using \(f\) for factor because these are the two terms that get multiplied with each other; the more general language is in part because of a desire to eventually make \(f_u\) and \(f_n\) multi-dimensional. polarity is assigned to both users and notes. the link between user ids and the underlying twitter accounts is intentionally kept hidden, but notes are public. in practice, the polarities generated by the algorithm, at least for the english-language data set, map very closely to the left vs right political spectrum. here are some examples of notes that have gotten polarities around -0.8: note polarity anti-trans rhetoric has been amplified by some conservative colorado lawmakers, including u.s. rep. lauren boebert, who narrowly won re-election in colorado's gop-leaning 3rd congressional district, which does not include colorado springs. https://coloradosun.com/2022/11/20/colorado-springs-club-q-lgbtq-trans/ -0.800 president trump explicitly undermined american faith in election results in the months leading up to the 2020 election. https://www.npr.org/2021/02/08/965342252/timeline-what-trump-told-supporters-for-months-before-they-attacked enforcing twitter's terms of service is not election interference. -0.825 the 2020 election was conducted in a free and fair manner. https://www.npr.org/2021/12/23/1065277246/trump-big-lie-jan-6-election -0.818 note that i am not cherry-picking here; these are literally the first three rows in the scored_notes.tsv spreadsheet generated by the algorithm when i ran it locally that have a polarity score (called corenotefactor1 in the spreadsheet) of less than -0.8. now, here are some notes that have gotten polarities around +0.8. it turns out that many of these are either people talking about brazilian politics in portuguese or tesla fans angrily refuting criticism of tesla, so let me cherry-pick a bit to find a few that are not: note polarity as of 2021 data, 64% of "black or african american" children lived in single-parent families. https://datacenter.aecf.org/data/tables/107-children-in-single-parent-families-by-race-and-ethnicity +0.809 contrary to rolling stones push to claim child trafficking is "a qanon adjacent conspiracy," child trafficking is a real and huge issue that this movie accurately depicts. operation underground railroad works with multinational agencies to combat this issue. https://ourrescue.org/ +0.840 example pages from these lgbtq+ children's books being banned can be seen here: https://i.imgur.com/8sy6cex.png these books are obscene, which is not protected by the us constitution as free speech. https://www.justice.gov/criminal-ceos/obscenity "federal law strictly prohibits the distribution of obscene matter to minors. +0.806 once again, it is worth reminding ourselves that the "left vs right divide" was not in any way hardcoded into the algorithm; it was discovered emergently by the calculation. this suggests that if you apply this algorithm in other cultural contexts, it could automatically detect what their primary political divides are, and bridge across those too. meanwhile, notes that get the highest helpfulness look like this. this time, because these notes are actually shown on twitter, i can just screenshot one directly: and another one: the second one touches on highly partisan political themes more directly, but it's a clear, high-quality and informative note, and so it gets rated highly. so all in all, the algorithm seems to work, and the ability to verify the outputs of the algorithm by running the code seems to work. what do i think of the algorithm? the main thing that struck me when analyzing the algorithm is just how complex it is. there is the "academic paper version", a gradient descent which finds a best fit to a five-term vector and matrix equation, and then the real version, a complicated series of many different executions of the algorithm with lots of arbitrary coefficients along the way. even the academic paper version hides complexity under the hood. the equation that it's optimizing is a degree-4 equation (as there's a degree-2 \(f_u * f_n\) term in the prediction formula, and compounding that the cost function measures error squared). while optimizing a degree-2 equation over any number of variables almost always has a unique solution, which you can calculate with fairly basic linear algebra, a degree-4 equation over many variables often has many solutions, and so multiple rounds of a gradient descent algorithm may well arrive at different answers. tiny changes to the input may well cause the descent to flip from one local minimum to another, significantly changing the output. the distinction between this, and algorithms that i helped work on such as quadratic funding, feels to me like a distinction between an economist's algorithm and an engineer's algorithm. an economist's algorithm, at its best, values being simple, being reasonably easy to analyze, and having clear mathematical properties that show why it's optimal (or least-bad) for the task that it's trying to solve, and ideally proves bounds on how much damage someone can do by trying to exploit it. an engineer's algorithm, on the other hand, is a result of iterative trial and error, seeing what works and what doesn't in the engineer's operational context. engineer's algorithms are pragmatic and do the job; economist's algorithms don't go totally crazy when confronted with the unexpected. or, as was famously said on a related topic by the esteemed internet philosopher roon (aka tszzl): of course, i would say that the "theorycel aesthetic" side of crypto is necessary precisely to distinguish protocols that are actually trustless from janky constructions that look fine and seem to work well but under the hood require trusting a few centralized actors or worse, actually end up being outright scams. deep learning works when it works, but it has inevitable vulnerabilities to all kinds of adversarial machine learning attacks. nerd traps and sky-high abstraction ladders, if done well, can be quite robust against them. and so one question i have is: could we turn community notes itself into something that's more like an economist algorithm? to give a view of what this would mean in practice, let's explore an algorithm i came up with a few years ago for a similar purpose: pairwise-bounded quadratic funding. the goal of pairwise-bounded quadratic funding is to plug a hole in "regular" quadratic funding, where if even two participants collude with each other, they can each contribute a very high amount of money to a fake project that sends the money back to them, and get a large subsidy that drains the entire pool. in pairwise quadratic funding, we assign each pair of participants a limited budget \(m\). the algorithm walks over all possible pairs of participants, and if the algorithm decides to add a subsidy to some project \(p\) because both participant \(a\) and participant \(b\) supported it, that subsidy comes out of the budget assigned to the pair \((a, b)\). hence, even if \(k\) participants were to collude, the amount they could steal from the mechanism is at most \(k * (k-1) * m\). an algorithm of exactly this form is not very applicable to the community notes context, because each user makes very few votes: on average, any two users would have exactly zero votes in common, and so the algorithm would learn nothing about users' polarities by just looking at each pair of users separately. the goal of the machine learning model is precisely to try to "fill in" the matrix from very sparse source data that cannot be analyzed in this way directly. but the challenge of this approach is that it takes extra effort to do it in a way that does not make the result highly volatile in the face of a few bad votes. does community notes actually fight polarization? one thing that we could do is analyze whether or not the community notes algorithm, as is, actually manages to fight polarization at all that is, whether or not it actually does any better than a naive voting algorithm. naive voting algorithms already fight polarization to some limited extent: a post with 200 upvotes and 100 downvotes does worse than a post that just gets the 200 upvotes. but does community notes do better than that? looking at the algorithm abstractly, it's hard to tell. why wouldn't a high-average-rating but polarizing post get a strong polarity and a high helpfulness? the idea is that polarity is supposed to "absorb" the properties of a note that cause it to get a lot of votes if those votes are conflicting, but does it actually do that? to check this, i ran my own simplified implementation for 100 rounds. the average results were: quality averages: group 1 (good): 0.30032841807271166 group 2 (good but extra polarizing): 0.21698871680927437 group 3 (neutral): 0.09443120045416832 group 4 (bad): -0.1521160965793673 in this test, "good" notes received a rating of +2 from users in the same political tribe and +0 from users in the opposite political tribe, and "good but extra polarizing" notes received a rating of +4 from same-tribe users and -2 from opposite-tribe users. same average, but different polarity. and it seems to actually be the case that "good" notes get a higher average helpfulness than "good but extra polarizing" notes. one other benefit of having something closer to an "economist's algorithm" would be having a clearer story for how the algorithm is penalizing polarization. how useful is this all in high-stakes situations? we can see some of how this works out by looking at one specific situation. about a month ago, ian bremmer complained that a highly critical community note that was added to a tweet by a chinese government official had been removed. the note, which is now no longer visible. screenshot by ian bremmer. this is heavy stuff. it's one thing to do mechanism design in a nice sandbox ethereum community environment where the largest complaint is $20,000 going to a polarizing twitter influencer. it's another to do it for political and geopolitical questions that affect many millions of people and where everyone, often quite understandably, is assuming maximum bad faith. but if mechanism designers want to have a significant impact into the world, engaging with these high-stakes environments is ultimately necessary. in the case of twitter, there is a clear reason why one might suspect centralized manipulation to be behind the note's removal: elon has a lot of business interests in china, and so there is a possibility that elon forced the community notes team to interfere with the algorithm's outputs and delete this specific one. fortunately, the algorithm is open source and verifiable, so we can actually look under the hood! let's do that. the url of the original tweet is https://twitter.com/mfa_china/status/1676157337109946369. the number at the end, 1676157337109946369, is the tweet id. we can search for that in the downloadable data, and identify the specific row in the spreadsheet that has the above note: here we get the id of the note itself, 1676391378815709184. we then search for that in the scored_notes.tsv and note_status_history.tsv files generated by running the algorithm. we get: the second column in the first output is the note's current rating. the second output shows the note's history: its current status is in the seventh column (needs_more_ratings), and the first status that's not needs_more_ratings that it received earlier on is in the fifth column (currently_rated_helpful). hence, we see that the algorithm itself first showed the note, and then removed it once its rating dropped somewhat seemingly no centralized intervention involved. we can see this another way by looking at the votes themselves. we can scan the ratings-00000.tsv file to isolate all the ratings for this note, and see how many rated helpful vs not_helpful: but if you sort them by timestamp, and look at the first 50 votes, you see 40 helpful votes and 9 not_helpful votes. and so we see the same conclusion: the note's initial audience viewed the note more favorably then the note's later audience, and so its rating started out higher and dropped lower over time. unfortunately, the exact story of how the note changed status is complicated to explain: it's not a simple matter of "before the rating was above 0.40, now it's below 0.40, so it got dropped". rather, the high volume of not_helpful replies triggered one of the outlier conditions, increasing the helpfulness score that the note needs to stay over the threshold. this is a good learning opportunity for another lesson: making a credibly neutral algorithm truly credible requires keeping it simple. if a note moves from being accepted to not being accepted, there should be a simple and legible story as to why. of course, there is a totally different way in which this vote could have been manipulated: brigading. someone who sees a note that they disapprove of could call upon a highly engaged community (or worse, a mass of fake accounts) to rate it not_helpful, and it may not require that many votes to drop the note from being seen as "helpful" to being seen as "polarized". properly minimizing the vulnerability of this algorithm to such coordinated attacks will require a lot more analysis and work. one possible improvement would be not allowing any user to vote on any note, but instead using the "for you" algorithmic feed to randomly allocate notes to raters, and only allow raters to rate those notes that they have been allocated to. is community notes not "brave" enough? the main criticism of community notes that i have seen is basically that it does not do enough. two recent articles that i have seen make this point. quoting one: the program is severely hampered by the fact that for a community note to be public, it has to be generally accepted by a consensus of people from all across the political spectrum. "it has to have ideological consensus," he said. "that means people on the left and people on the right have to agree that that note must be appended to that tweet." essentially, it requires a "cross-ideological agreement on truth, and in an increasingly partisan environment, achieving that consensus is almost impossible, he said. this is a difficult issue, but ultimately i come down on the side that it is better to let ten misinformative tweets go free than it is to have one tweet covered by a note that judges it unfairly. we have seen years of fact-checking that is brave, and does come from the perspective of "well, actually we know the truth, and we know that one side lies much more often than the other". and what happened as a result? honestly, some pretty widespread distrust of fact-checking as a concept. one strategy here is to say: ignore the haters, remember that the fact checking experts really do know the facts better than any voting system, and stay the course. but going all-in on this approach seems risky. there is value in building cross-tribal institutions that are at least somewhat respected by everyone. as with william blackstone's dictum and the courts, it feels to me that maintaining such respect requires a system that commits far more sins of omission than it does sins of commission. and so it seems valuable to me that there is at least one major organization that is taking this alternate path, and treating its rare cross-tribal respect as a resource to be cherished and built upon. another reason why i think it is okay for community notes to be conservative is that i do not think it is the goal for every misinformative tweet, or even most misinformative tweets, to receive a corrective note. even if less than one percent of misinformative tweets get a note providing context or correcting them, community notes is still providing an exceedingly valuable service as an educational tool. the goal is not to correct everything; rather, the goal is to remind people that multiple perspectives exist, that certain kinds of posts that look convincing and engaging in isolation are actually quite incorrect, and you, yes you, can often go do a basic internet search to verify that it's incorrect. community notes cannot be, and is not meant to be, a miracle cure that solves all problems in public epistemology. whatever problems it does not solve, there is plenty of room for other mechanisms, whether newfangled gadgets such as prediction markets or good old-fashioned organizations hiring full-time staff with domain expertise, to try to fill in the gaps. conclusions community notes, in addition to being a fascinating social media experiment, is also an instance of a fascinating new and emerging genre of mechanism design: mechanisms that intentionally try to identify polarization, and favor things that bridge across divides rather than perpetuate them. the two other things in this category that i know about are (i) pairwise quadratic funding, which is being used in gitcoin grants and (ii) polis, a discussion tool that uses clustering algorithms to help communities identify statements that are commonly well-received across people who normally have different viewpoints. this area of mechanism design is valuable, and i hope that we can see a lot more academic work in this field. algorithmic transparency of the type that community notes offers is not quite full-on decentralized social media if you disagree with how community notes works, there's no way to go see a view of the same content with a different algorithm. but it's the closest that very-large-scale applications are going to get within the next couple of years, and we can see that it provides a lot of value already, both by preventing centralized manipulation and by ensuring that platforms that do not engage in such manipulation can get proper credit for doing so. i look forward to seeing both community notes, and hopefully many more algorithms of a similar spirit, develop and grow over the next decade. dark mode toggle trust models 2020 aug 20 see all posts one of the most valuable properties of many blockchain applications is trustlessness: the ability of the application to continue operating in an expected way without needing to rely on a specific actor to behave in a specific way even when their interests might change and push them to act in some different unexpected way in the future. blockchain applications are never fully trustless, but some applications are much closer to being trustless than others. if we want to make practical moves toward trust minimization, we want to have the ability to compare different degrees of trust. first, my simple one-sentence definition of trust: trust is the use of any assumptions about the behavior of other people. if before the pandemic you would walk down the street without making sure to keep two meters' distance from strangers so that they could not suddenly take out a knife and stab you, that's a kind of trust: both trust that people are very rarely completely deranged, and trust that the people managing the legal system continue to provide strong incentives against that kind of behavior. when you run a piece of code written by someone else, you trust that they wrote the code honestly (whether due to their own sense of decency or due to an economic interest in maintaining their reputations), or at least that there exist enough people checking the code that a bug would be found. not growing your own food is another kind of trust: trust that enough people will realize that it's in their interests to grow food so they can sell it to you. you can trust different sizes of groups of people, and there are different kinds of trust. for the purposes of analyzing blockchain protocols, i tend to break down trust into four dimensions: how many people do you need to behave as you expect? out of how many? what kinds of motivations are needed for those people to behave? do they need to be altruistic, or just profit seeking? do they need to be uncoordinated? how badly will the system fail if the assumptions are violated? for now, let us focus on the first two. we can draw a graph: the more green, the better. let us explore the categories in more detail: 1 of 1: there is exactly one actor, and the system works if (and only if) that one actor does what you expect them to. this is the traditional "centralized" model, and it is what we are trying to do better than. n of n: the "dystopian" world. you rely on a whole bunch of actors, all of whom need to act as expected for everything to work, with no backups if any of them fail. n/2 of n: this is how blockchains work they work if the majority of the miners (or pos validators) are honest. notice that n/2 of n becomes significantly more valuable the larger the n gets; a blockchain with a few miners/validators dominating the network is much less interesting than a blockchain with its miners/validators widely distributed. that said, we want to improve on even this level of security, hence the concern around surviving 51% attacks. 1 of n: there are many actors, and the system works as long as at least one of them does what you expect them to. any system based on fraud proofs falls into this category, as do trusted setups though in that case the n is often smaller. note that you do want the n to be as large as possible! few of n: there are many actors, and the system works as long as at least some small fixed number of them do what you expect them do. data availability checks fall into this category. 0 of n: the systems works as expected without any dependence whatsoever on external actors. validating a block by checking it yourself falls into this category. while all buckets other than "0 of n" can be considered "trust", they are very different from each other! trusting that one particular person (or organization) will work as expected is very different from trusting that some single person anywhere will do what you expect them to. "1 of n" is arguably much closer to "0 of n" than it is to "n/2 of n" or "1 of 1". a 1-of-n model might perhaps feel like a 1-of-1 model because it feels like you're going through a single actor, but the reality of the two is very different: in a 1-of-n system, if the actor you're working with at the moment disappears or turns evil, you can just switch to another one, whereas in a 1-of-1 system you're screwed. particularly, note that even the correctness of the software you're running typically depends on a "few of n" trust model to ensure that if there's bugs in the code someone will catch them. with that fact in mind, trying really hard to go from 1 of n to 0 of n on some other aspect of an application is often like making a reinforced steel door for your house when the windows are open. another important distinction is: how does the system fail if your trust assumption is violated? in blockchains, two most common types of failure are liveness failure and safety failure. a liveness failure is an event in which you are temporarily unable to do something you want to do (eg. withdraw coins, get a transaction included in a block, read information from the blockchain). a safety failure is an event in which something actively happens that the system was meant to prevent (eg. an invalid block gets included in a blockchain). here are a few examples of trust models of a few blockchain layer 2 protocols. i use "small n" to refer to the set of participants of the layer 2 system itself, and "big n" to refer to the participants of the blockchain; the assumption is always that the layer 2 protocol has a smaller community than the blockchain itself. i also limit my use of the word "liveness failure" to cases where coins are stuck for a significant amount of time; no longer being able to use the system but being able to near-instantly withdraw does not count as a liveness failure. channels (incl state channels, lightning network): 1 of 1 trust for liveness (your counterparty can temporarily freeze your funds, though the harms of this can be mitigated if you split coins between multiple counterparties), n/2 of big-n trust for safety (a blockchain 51% attack can steal your coins) plasma (assuming centralized operator): 1 of 1 trust for liveness (the operator can temporarily freeze your funds), n/2 of big-n trust for safety (blockchain 51% attack) plasma (assuming semi-decentralized operator, eg. dpos): n/2 of small-n trust for liveness, n/2 of big-n trust for safety optimistic rollup: 1 of 1 or n/2 of small-n trust for liveness (depends on operator type), n/2 of big-n trust for safety zk rollup: 1 of small-n trust for liveness (if the operator fails to include your transaction, you can withdraw, and if the operator fails to include your withdrawal immediately they cannot produce more batches and you can self-withdraw with the help of any full node of the rollup system); no safety failure risks zk rollup (with light-withdrawal enhancement): no liveness failure risks, no safety failure risks finally, there is the question of incentives: does the actor you're trusting need to be very altruistic to act as expected, only slightly altruistic, or is being rational enough? searching for fraud proofs is "by default" slightly altruistic, though just how altruistic it is depends on the complexity of the computation (see the verifier's dilemma), and there are ways to modify the game to make it rational. assisting others with withdrawing from a zk rollup is rational if we add a way to micro-pay for the service, so there is really little cause for concern that you won't be able to exit from a rollup with any significant use. meanwhile, the greater risks of the other systems can be alleviated if we agree as a community to not accept 51% attack chains that revert too far in history or censor blocks for too long. conclusion: when someone says that a system "depends on trust", ask them in more detail what they mean! do they mean 1 of 1, or 1 of n, or n/2 of n? are they demanding these participants be altruistic or just rational? if altruistic, is it a tiny expense or a huge expense? and what if the assumption is violated do you just need to wait a few hours or days, or do you have assets that are stuck forever? depending on the answers, your own answer to whether or not you want to use that system might be very different. microsoft to sponsor ethereum’s devcon1 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search microsoft to sponsor ethereum’s devcon1 posted by george hallam on october 28, 2015 events london, united kingdom, october 28th ethereum is happy to announce that microsoft will sponsor and attend dξvcon1, the premier ethereum developer conference. marley gray, director of technology strategy us financial services at microsoft said, "microsoft is excited to sponsor and attend ethereum's devcon1. we find the ethereum blockchain incredibly powerful and look forward to collaborating within the ethereum community. we see a future where the combination of microsoft azure and ethereum can enable new innovative platforms like blockchain-as-a-service. this will serve as an inflection point to bring blockchain technology to enterprise clientele". ethereum founder vitalik buterin notes “dξvcon1 is very excited to work with microsoft and we look forward to having them in london.” dξvcon1 will held november 9th-13th at gibson hall in london, uk. it will feature five days of technical presentations and events on research and core protocols, decentralized application (dapp) development, and industry and social implications. ethereum is a decentralised blockchain protocol that executes smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference. ethereum is how the internet was supposed to work. ### previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum dev update 2015 / week 41 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum dev update 2015 / week 41 posted by taylor gerring on october 13, 2015 research & development in an effort to bring the community more information about the going-ons at ethereum, we're planning to release semi-weekly updates of project progress. i hope this provides some high-level information without having to live on github ;) the go and c++ teams have daily standups and are making regular progress along with several other improvements around the ecosystem. concurrent with this, the teams are gearing up for devcon1, preparing talks and presentations along with finalizing more details in the coming weeks. if you haven't already registered, visit https://devcon.ethereum.org/ to join the fun next month in london. now on to the updates... mist hoping to have gui wallet next week with chocie of geth or eth client. the team is spending time polishing remaining bugs and bundling binaries for the various platforms. light client the les subprotocol being developed and prototyped into the clients. first release is only a start and work will continue on light-client features into the future, more robustly supporting mobile and embedded scenarios. separately, work on fast-syncing will help aid in this transition. solidity/mix improvements to passing certain types in solidity has been added along with improving exception handling throughout. multiple improvements have been pushed to christian's browser-solidity project. continued refinement is also being put into the mix ui. cpp-ethereum the team is currently working towards 1.0 release and hoping to have new rc available soon. refactoring of the error reporting mechanism ("exceptions") and the ability for libraries to access storage of the calling contract by passing storage reference types is now possible. additionally, the build system being overhauled with a switch to jenkins. go-etheruem continuing working towards geth 1.3 release, which will consist primarily of internal api improvements and speedups in preparation for future features. more work being put into windows builds, hoping to have source-built scripts available soon. fast-sync code being added to client, allowing full-nodes to syncronize to the network much quicker. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum in practice part 2: how to build a better democracy in under a 100 lines of code | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum in practice part 2: how to build a better democracy in under a 100 lines of code posted by alex van de sande on december 4, 2015 research & development this is a second post on a series on how to use the current ethereum tools to build smart contracts without needing programming experience. download the latest version of the ethereum wallet, read the first part and let's have fun! creating a token is fun, but what is the value of a token that doesn't do anything new? we are now going to create a new contract that uses the tokens we just created. the contract will be a democratic organization that lives on the blockchain and that anyone holding a share token will be able to vote on proposals. so let's go back to "contracts" and then "deploy contract" and paste the dao source code on the "solidity source" field. choose the contract "democracy" on the picker and then select these parameters: on the amount field you can add any ether amount you want your dao to start with. since you can send ether to it at any time in the future, if this is the first time you've been doing this then keep the amount at 0 and send the money later. on the sharesaddress field, paste the address of the token contract you just created. pay attention to the icon and color of the little circle that appears by the side of the address. if it doesn’t match exactly the one for the contract you created previously, then there's an error. on minimumsharesforvoting pick what is the minimum quorum of shareholders that need to vote on an issue before it passes. here you must put the integer number of the minimum token possible, so if you created a token with 2 decimal places, putting 500 here will mean that in order for a proposal to be executed then the number of votes must be more than 5% of the total shares of the company. minutesfordebating: this is the minimum time a proposal must be discussed and voted on before the results can be tallied up. put a small number like 10 minutes if you want just to create something for testing, but put something like 20,000 if you want to store large amounts of ether, so all proposals must stay there for at least two weeks.   your contract should be looking something like this:   after a few seconds you'll be redirected to a the dashboard where you'll see your new contract being created: you can also see the link to the new contract on the "contracts" tab. if it's not there (for example: if you are adding a contract that you didn't create) then you'll have to add it manually. to add that you'll need the contract's address and it's json interface, a string of text that explains to the wallet how to interact with the code. you can pick any name you want and press ok. click your new contract box and you'll see a contract page. if you haven't already deposited some ether on that contract, click on deposit and put some small amount of ether so you can test how this particular democracy works. if the contract already has some funds, click the "show contract info" button. on the "read from contract" you can see all the functions you can execute for free on the contract, as they are just reading information from the blockchain. here you can see, for instance, that the "sharestokenaddress" is indeed the correct address or that there are currently 0 proposals on the contract. let's change that. on the "write to contract" you have a list of all the functions that will attempt to do some computation that saves data to the blockchain, and therefore will cost ether. select "newproposal" and it will show all the options options for that function.   for "beneficiary" add the address of someone you want to send ether to, then put how many ethers you want on the "etheramount" (must be an integer) and finally some text describing the reason you want to do this. leave transactionbytecode blank for now. click execute and type your password. after a few seconds the numproposals will increase to 1 and the first proposal, number 0, will appear on the left column. as you add more proposals, you can see any of them by simply putting the proposal number on the "proposals" field and you can read all about it. voting on a proposal is also very simple. choose "vote" on the function picker. type the proposal number on the first box and check the "yes" box if you agree with it (or leave it blank to vote against it). click "execute" to send your vote.   when the minimum voting time has passed, you can select "executeproposal". if the proposal was simply sending ether, then you can also leave the "transactionbytecode" field blank. after pressing "execute" but before typing your password, pay attention to the screen that appears. if there is a warning on the "estimated fee consumption" field, then this means that for some reason the function called will not execute and will be abruptly terminated. it can mean many things, but in the context of this contract this warning will show up whenever you try to execute a contract before its deadline has passed, or if the user is trying to send a different bytecode data than the original proposal had. for security reasons if any of these things happens, the contract execution is abruptly terminated and the user that attempted the illegal transaction will lose the all the ether he sent to pay transaction fees. if everything went well you should be able to see the results of the vote in a few seconds. the "opentovote" parameter on the first box will turn to false while the proposalpassed will reflect if the proposal has been accepted or not. you should also be able to see that the ether balance of the contract will go down and the equivalent ether will be sent to the beneficiary of the ether you wanted to send. now take a moment to let that in: you just created an organization that only exists on the blockchain, that obeys votes based on completely digital tokens, but yet it can move real value around and create a very real impact on the world. also notice that the organization is not under your control anymore: it will execute only the exact code you used to create it, forever. you can't bribe it, you can't earmark it and the same rules apply either you are moving 0.01 or 1,000,000 ethers. can it get any better than this? actually, it can. on our next post we will explore how you can use "transactionbytecode" to allow the dao to execute any kind of ethereum transaction, even owning or creating other contracts. we'll also modify the token code to allow the dao to control the amount of a token that exists on circulation and how to send it forward. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements security alert – [previous security patch can lead to invalid state root on go clients with a specific transaction sequence – fixed. please update.] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security alert – [previous security patch can lead to invalid state root on go clients with a specific transaction sequence – fixed. please update.] posted by jutta steiner on september 10, 2015 security   summary: implementation bug in the go client may lead to invalid state affected client versions: latest (unpatched) versions of go client; v1.1.2, v1.0.4 tags and develop, master branches before september 9. likelihood: low severity: high impact: high details: go ethereum client does not correctly restore state of execution environment when a transaction goes out-of-gas if within the same block a contract was suicided. this would result in an invalid copy operation of the state object; flagging the contract as not deleted. this operation would cause a consensus issue between the other implementations.   effects on expected chain reorganisation depth: none remedial action taken by ethereum: provision of hotfixes as below. proposed temporary workaround: use python or c++ client   if using the ppa: sudo apt-get update then sudo apt-get upgrade if using brew: brew update then brew reinstall ethereum if using a windows binary: download the updated binary from https://github.com/ethereum/go-ethereum/releases/tag/v1.1.3   master branch commit: https://github.com/ethereum/go-ethereum/commit/9ebe787d3afe35902a639bf7c1fd68d1e591622a   if you’re building from source: git fetch origin && git checkout origin/master followed by a make geth previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements slasher: a punitive proof-of-stake algorithm | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search slasher: a punitive proof-of-stake algorithm posted by vitalik buterin on january 15, 2014 research & development the purpose of this post is not to say that ethereum will be using slasher in place of dagger as its main mining function. rather, slasher is a useful construct to have in our war chest in case proof of stake mining becomes substantially more popular or a compelling reason is provided to switch. slasher may also benefit other cryptocurrencies that wish to exist independently of ethereum. special thanks to tacotime for some inspiration, and for jack walker for improvement suggestions. proof of stake mining has for a long time been a large area of interest to the cryptocurrency community. the first proof-of-stake based coin, ppcoin, was releasd by sunny king in 2012, and has consistently remained among the top five alternative currencies by monetary base since then. and for good reason; proof of stake has a number of advantages over proof of work as a mining method. first of all, proof of stake is much more environmentally friendly; while proof of work requires miners to effectively burn computational power on useless calculations to secure the network, proof of stake effectively simulates the burning, so no real-world energy or resources are ever actually wasted. second, there are centralization concerns. with proof of work, mining has been essentially dominated by specialized hardware (“application-specific integrated circuits” / asics), and there is a large risk that a single large player such as intel or a major bank will take over and de-facto monopolize the market. memory-hard mining algorithms like scrypt and now dagger mitigate this to a large extent, but even still not perfectly. once again, proof of stake, if it can be made to work, is essentially a perfect solution. however, proof of stake, as implemented in nearly every currency so far, has one fundamental flaw: as one prominent bitcoin developer put it, “there’s nothing at stake”. the meaning of the statement becomes clear when we attempt to analyze what exactly is going on in the event of an attempted 51% attack, the situation that any kind of proof-of-work like mechanism is intended to prevent. in a 51% attack, an attacker a sends a transaction from a to b, waits for the transaction to be confirmed in block k1 (with parent k), collects a product from b, and then immediately creates another block k2 on top of k – with a transaction sending the same bitcoins but this time from a to a. at that point, there are two blockchains, one from block k1 and another from block k2. if b can add blocks on top of k2 faster than the entire legitimate network can create blocks on top of k1, the k2 blockchain will win – and it will be as if the payment from a to b had never happened. the point of proof of work is to make it take a certain amount of computational power to create a block, so that in order for k2 to outrace k1 b would have to have more computational power than the entire legitimate network combined. in the case of proof of stake, it doesn’t take computational power to create a work – instead, it takes money. in ppcoin, every “coin” has a chance per second of becoming the lucky coin that has the right to create a new valid block, so the more coins you have the faster you can create new blocks in the long run. thus, a successful 51% attack, in theory, requires not having more computing power than the legitimate network, but more money than the legitimate network. but here we see the difference between proof of work and proof of stake: in proof of work, a miner can only mine on one fork at a time, so the legitimate network will support the legitimate blockchain and not an attacker’s blockchain. in proof of stake, however, as soon as a fork happens miners will have money in both forks at the same time, and so miners will be able to mine on both forks. in fact, if there is even the slightest chance that the attack will succeed, miners have the incentive to mine on both. if a miner has a large number of coins, the miner will want to oppose attacks to preserve the value of their own coins; in an ecosystem with small miners, however, network security potentially falls apart in a classic public goods problem as no single miner has substantial impact on the result and so every miner will act purely “selfishly”. the solution some have theorized that the above argument is a deathblow to all proof of stake, at least without a proof of work component assisting it. and in a context where every chain is only aware of itself, this is indeed provably true. however, there is actually one clever way to get around the issue, and one which has so far been underexplored: make the chain aware of other chains. then, if a miner is caught mining on two chains at the same time, that miner can be penalized. however, it is not at all obvious how to do this with a ppcoin-like design. the reason is this: mining is a random process. that is to say, a miner with 0.1% of the stake has a 0.1% chance of mining a valid block on block k1, and a 0.1% chance of mining a valid block on block k2, but only a 0.0001% chance of mining a valid block on both. and in that case, the miner can simply hold back the second block – because mining is probabilistic, the miner can still gain 99.9% of the benefit of mining on the second chain. the following proposal, however, outlines an algorithm, which we are calling slasher to express its harshly punitive nature, for avoiding this proposal. the design description given here uses address balances for clarity, but can easily be used to work with “unspent transaction outputs”, or any other similar abstraction that other currencies may use. blocks are mined with proof of work. however, we make one modification. when creating a block k, a miner must include the value h(n) for some random n generated by the miner. the miner must claim the reward by releasing a transaction uncovering n between block k+100 and k+900. the proof of work reward is very low, ideally encouraging energy usage equal to about 1% of that of bitcoin. the target block time is 30 seconds. suppose the total money supply is m, and n[i] is the n value at block i. at block k+1000, an address a with balance b gains a “signing privilege” if sha256(n[k] + n[k+1] + ... + n[k+99] + a) < 2^256 * 64 * b / m. essentially, an address has a chance of gaining a signing privilege proportional to the amount of money that it has, and on average 64 signing privileges will be assigned each block. at block k+2000, miners with signing privileges from block k have the opportunity to sign the block. the number of signatures is what determines the total length of one blockchain versus another. a signature awards the signer a reward that is substantially larger than the proof of work reward, and this reward will unlock by block k+3000. suppose that a user detects two signatures made by address a on two distinct blocks with height k+2000. that node can then publish a transaction containing those two signatures, and if that transaction is included before block k+3000 it destroys the reward for that signature and sends 33% to the user that ratted the cheater out. the key to this design is how the signing privileges are distributed: instead of the signing privilege being randomly based on the previous block, the signing privilege is based on the block two thousand blocks ago. thus, in the event of a fork, a miner that gets lucky in one chain will also get lucky in the other, completely eliminating the probabilistic dual-mining attack that is possible with ppcoin. another way of looking at it is that because slasher uses proof-of-stake-2000-blocks-ago instead of proof-of-stake now, and forks will almost certainly not last 2000 blocks, there is only one currency supply to mine with, so there is indeed “something at stake”. the penalty of block reward loss ensures that every node will take care to sign only one block at each block number. the use of 100 pre-committed random numbers is an idea taken from provably fair gambling protocols; the idea is that powerful miners have no way of attempting to create many blocks and publishing only those that assign their own stake a signing privilege, since they do not know what any of the other random data used to determine the stakeholder is when they create their blocks. the system is not purely proof-of-stake; some minimal proof-of-work will be required to maintain a time interval between blocks. however, a 51% attack on the proof of work would be essentially inconsequential, as proof of stake signing is the sole deciding factor in which blockchain wins. furthermore, the energy usage from proof of work can be made to be 95-99% lower, resolving the environmental concern with proof of work. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements more thoughts on scripting and future-compatibility | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search more thoughts on scripting and future-compatibility posted by vitalik buterin on february 5, 2014 research & development my previous post introducing ethereum script 2.0 was met with a number of responses, some highly supportive, others suggesting that we switch to their own preferred stack-based / assembly-based / functional paradigm, and offering various specific criticisms that we are looking hard at. perhaps the strongest criticism this time came from sergio damian lerner, bitcoin security researcher, developer of qixcoin and to whom we are grateful for his analysis of dagger. sergio particularly criticizes two aspects of the change: the fee system, which is changed from a simple one-variable design where everything is a fixed multiple of the basefee, and the loss of the crypto opcodes. the crypto opcodes are the more important part of sergio’s argument, and i will handle that issue first. in ethereum script 1.0, the opcode set had a collection of opcodes that are specialized around certain cryptographic functions – for example, there was an opcode sha3, which would take a length and a starting memory index off the stack and then push the sha3 of the string taken from the desired number of blocks in memory starting from the starting index. there were similar opcodes for sha256and ripemd160 and there were also crypto opcodes oriented around secp256k1 elliptic curve operations. in es2, those opcodes are gone. instead, they are replaced by a fluid system where people will need to write sha256 in es manually (in practice, we would offer a commision or bounty for this), and then later on smart interpreters can seamlessly replace the sha256 es script with a plain old machine-code (or even hardware) version of sha256 of the sort that you use when you call sha256 in c++. from an outside view, es sha256 and machine code sha256 are indistinguishable; they both compute the same function and therefore make the same transformations to the stack, the only difference is that the latter is hundreds of times faster, giving us the same efficiency as if sha256 was an opcode. a flexible fee system can then also be implemented to make sha256 cheaper to accommodate its reduced computation time, ideally making it as cheap as an opcode is now. sergio, however, prefers a different approach: coming with lots of crypto opcodes out of the box, and using hard-forking protocol changes to add new ones if necessary further down the line. he writes: first, after 3 years of watching bitcoin closely i came to understand that a cryptocurrency is not a protocol, nor a contract, nor a computer-network. a cryptocurrency is a community. with the exception of a very few set of constants, such as the money supply function and the global balance, anything can be changed in the future, as long as the change is announced in advance. bitcoin protocol worked well until now, but we know that in the long term it will face scalability issues and it will need to change accordingly. short term benefits, such as the simplicity of the protocol and the code base, helped the bitcoin get worldwide acceptance and network effect. is the reference code of bitcoin version 0.8 as simple as the 0.3 version? not at all. now there are caches and optimizations everywhere to achieve maximum performance and higher dos security, but no one cares about this (and nobody should). a cryptocurrency is bootstrapped by starting with a simple value proposition that works in the short/mid term. this is a point that is often brought up with regard to bitcoin. however, the more i look at what is actually going on in bitcoin development, the more i become firmly set in my position that, with the exception of very early-stage cryptographic protocols that are in their infancy and seeing very low practical usage, the argument is absolutely false. there are currently many flaws in bitcoin that can be changed if only we had the collective will to. to take a few examples: the 1 mb block size limit. currently, there is a hard limit that a bitcoin block cannot have more than 1 mb of transactions in it – a cap of about seven transactions per second. we are starting to brush against this limit already, with about 250 kb in each block, and it’s putting pressure on transaction fees already. in most of bitcoin’s history, fees were around $0.01, and every time the price rose the default btc-denominated fee that miners accept was adjusted down. now, however, the fee is stuck at $0.08, and the developers are not adjusting it down arguably because adjusting the fee back down to $0.01 would cause the number of transactions to brush against the 1 mb limit. removing this limit, or at the very least setting it to a more appropriate value like 32 mb, is a trivial change; it is only a single number in the source code, and it would clearly do a lot of good in making sure that bitcoin continues to be used in the medium term. and yet, bitcoin developers have completely failed to do it. the op_checkmultisig bug. there is a well-known bug in the op_checkmultisig operator, used to implement multisig transactions in bitcoin, where it requires an additional dummy zero as an argument which is simply popped off the stack and not used. this is highly non-intuitive, and confusing; when i personally was working on implementing multisig for pybitcointools, i was stuck for days trying to figure out whether the dummy zero was supposed to be at the front or take the place of the missing public key in a 2-of-3 multisig, and whether there are supposed to be two dummy zeroes in a 1-of-3 multisig. eventually, i figured it out, but i would have figured it out much faster had the operation of theop_checkmultisig operator been more intuitive. and yet, the bug has not been fixed. the bitcoind client. the bitcoind client is well-known for being a very unwieldy and non-modular contraption; in fact, the problem is so serious that everyone looking to build a bitcoind alternative that is more scalable and enterprise-friendly is not using bitcoind at all, instead starting from scratch. this is not a core protocol issue, and theoretically changing the bitcoind client need not involve any hard-forking changes at all, but the needed reforms are still not being done. all of these problems are not there because the bitcoin developers are incompetent. they are not; in fact, they are very skilled programmers with deep knowledge of cryptography and the database and networking issues inherent in cryptocurrency client design. the problems are there because the bitcoin developers very well realize that bitcoin is a 10-billion-dollar train hurtling along at 400 kilometers per hour, and if they try to change the engine midway through and even the tiniest bolt comes loose the whole thing could come crashing to a halt. a change as simple as swapping the database back in march 2011 almost did. this is why in my opinion it is irresponsible to leave a poorly designed, non-future-proof protocol, and simply say that the protocol can be updated in due time. on the contrary, the protocol must be designed to have an appropriate degree of flexibility from the start, so that changes can be made by consensus to automatically without needing to update any software. now, to address sergio’s second issue, his main qualm with modifiable fees: if fees can go up and down, it becomes very difficult for contracts to set their own fees, and if a fee goes up unexpectedly then that may open up a vulnerability through which an attacker may even be able to force a contract to go bankrupt. i must thank sergio for making this point; it is something that i had not yet sufficiently considered, and we will need to think carefully about when making our design. however, his solution, manual protocol updates, is arguably no better; protocol updates that change fee structures can expose new economic vulnerabilities in contracts as well, and they are arguably even harder to compensate for because there are absolutely no restrictions on what content manual protocol updates can contain. so what can we do? first of all, there are many intermediate solutions between sergio’s approach – coming with a limited fixed set of opcodes that can be added to only with a hard-forking protocol change – and the idea i provided in the es2 blogpost of having miners vote on fluidly changing fees for every script. one approach might be to make the voting system more discrete, so that there would be a hard line between a script having to pay 100% fees and a script being “promoted” to being an opcode that only needs to pay a 20x cryptofee. this could be done via some combination of usage counting, miner voting, ether holder voting or other mechanisms. this is essentially a built-in mechanism for doing hardforks that does not technically require any source code updates to apply, making it much more fluid and non-disruptive than a manual hardfork approach. second, it is important to point out once again that the ability to efficiently do strong crypto is not gone, even from the genesis block; when we launch ethereum, we will create a sha256 contract, a sha3 contract, etc and “premine” them into pseudo-opcode status right from the start. so ethereum will come with batteries included; the difference is that the batteries will be included in a way that seamlessly allows for the inclusion of more batteries in the future. but it is important to note that i consider this ability to add in efficient optimized crypto ops in the future to be mandatory. theoretically, it is possible to have a “zerocoin” contract inside of ethereum, or a contract using cryptographic proofs of computation (scip) and fully homomorphic encryption so you can actually use ethereum as the “decentralized amazon ec2 instance” for cloud computing that many people now incorrectly believe it to be. once quantum computing comes out, we might need to move to contracts that rely on ntru; one sha4 or sha5 come out we might need to move to contracts that rely on them. once obfuscation technology matures, contracts will want to rely on that to store private data. but in order for all of that to be possible with anything less than a $30 fee per transaction, the underlying cryptography would need to be implemented in c++ or machine code, and there would need to be a fee structure that reduces the fee for the operations appropriately once the optimizations have been made. this is a challenge to which i do not see any easy answers, and comments and suggestions are very much welcome. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements security alert [implementation of blockhash instruction in c++ and go clients can potentially cause consensus issue – fixed. please update.] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security alert [implementation of blockhash instruction in c++ and go clients can potentially cause consensus issue – fixed. please update.] posted by gustav simonsson on october 22, 2015 security summary: erroneous implementation of blockhash can trigger a chain reorganisation leading to consensus problems affected configurations: all geth versions up to 1.1.3 and 1.2.2. all eth versions prior to 1.0.0. likelihood: low severity: medium impact: medium details: both c++ (eth) and go (geth) clients have an erroneous implementation of an edge case in the ethereum virtual machine, specifically which chain the blockhash instruction uses for retrieving a block hash. this edge case is very unlikely to happen on a live network as it would only be triggered in certain types of chain reorganisations (a contract executing blockhash(n 1) where n is the head of a non-canonical subchain that is not-yet reorganised to become the canonical (best/longest) chain but will be after the block is processed). pyethereum is unaffected. effects on expected chain reorganisation depth: none remedial action taken by ethereum: provision of hotfixes as below. geth: ppa: sudo apt-get update then sudo apt-get upgrade brew: brew update then brew reinstall ethereum windows: download the updated binary from https://github.com/ethereum/go-ethereum/releases/tag/v1.2.3 building from source: git fetch origin && git checkout origin/master make geth master branch commit: e55594ca0e131d518944e98701fc067735b11152 eth: ppa: https://gavofyork.gitbooks.io/turboethereum/content/chapter1.html window and mac os x:  https://build.ethdev.com/cpp-binaries-data/ source: https://github.com/ethereum/webthree-umbrella/tree/release building from source: https://github.com/ethereum/webthree-umbrella/wiki previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle farklı zk-evm türleri 2022 aug 04 see all posts pse, polygon hermez, zksync, scroll, matter labs ve starkware ekiplerine münazara ve inceleme, çeviri için eren'e özel teşekkürler. son zamanlarda birçok "zk-evm" projesinin çarpıcı duyurular yaptığı görüldü. polygon zk-evm projesini açık kaynaklı hale getirdi, zksync, zksync 2.0 için planlarını yayınladı ve görece yeni bir oyuncu olan scroll, zk-evm'lerini duyurdu. ayrıca privacy and scaling explorations ekibinin, nicolas liochon ve arkadaşlarının ekibinin, evm'den starkware'ın zk-dostu dili cairo'ya yönelik bir alfa derleyici ve kaçırdığım en az birkaç kişinin daha kesinlikle devam eden çabaları da var. tüm bu projelerin temel hedefi aynı: zk-snark teknolojisini kullanarak ethereum-benzeri işlemlerin kriptografik yürütülme kanıtlarını oluşturmak, bu kanıtları ethereum zincirini doğrulamayı çok daha kolay hale getirmek veya ethereum'un sunduğuna (neredeyse) eşdeğer ancak çok daha ölçeklenebilir olan zk-rollup'lar inşa etmek için kullanmaktır. ancak bu projeler arasında ince farklılıklar bulunmakta ve bunlar pratiklik ile hız arasındaki değiş tokuşları nasıl yaptıklarını belirlemektedir. bu yazı, evm denkliğinin farklı "tiplerinin" bir taksonomisini ve her bir tipe ulaşmaya çalışmanın fayda ve maliyetlerinin neler olduğunu açıklamaya çalışacaktır. genel bakış (grafik biçiminde) tip 1 (tamamen ethereum-dengi) tip 1 zk-evm'ler, tamamen ve ödün vermeden ethereum-dengi olmaya çalışırlar. kanıtları oluşturmayı kolaylaştırmak için ethereum sisteminin herhangi bir parçasını değiştirmezler. ne kadar çevresel olursa olsun hash'leri, state tree'leri, transaction tree'leri, ön-derlemeleri veya başka herhangi bir konsensüse dahil mantığı değiştirmezler. avantaj: mükemmel uyumluluk hedef, ethereum bloklarını bugünkü haliyle doğrulayabilmektir veya en azından yürütme-katmanı tarafını doğrulayabilmektir (bu nedenle, beacon zinciri konsensüs mantığı dahil değil, ancak tüm işlem yürütme ve akıllı kontrat ve hesap mantığı dahil). tip 1 zk-evm'ler, ethereum katman 1'inin daha ölçeklenebilir hale gelmesini sağlamak için sonunda ihtiyacımız olan şeydir. uzun vadede, tip 2 veya tip 3 zk-evm'lerde test edilmiş ethereum'a yapılan modifikasyonlar, ethereum'a uygun bir şekilde tanıtılabilir, ancak böyle bir yeniden tasarlama kendi komplekslikleri ile gelir. tip 1 zk-evm'ler aynı zamanda rollup'lar için idealdir, çünkü rollup'ların birçok altyapıyı tekrar kullanmalarına izin verirler. örneğin, ethereum execution istemcileri, rollup blokları oluşturmak ve işlemek için olduğu gibi kullanılabilir (veya en azından para çekme işlemleri implement edildiğinde ve bu işlevsellik, rollup'a eth yatırılmasını desteklemek için tekrar kullanılabilir), bu nedenle blok gezginleri, blok üretimi vb. gibi araçlar çok kolay bir şekilde tekrar kullanılabilir. dezavantaj: kanıtlayıcı zamanı ethereum, başlangıçta zk-dostu olacak şekilde tasarlanmamıştı, bu nedenle ethereum protokolünün, zk-kanıtlama için büyük miktarda hesaplama gerektiren birçok parçası var. tip 1, ethereum'u tam olarak kopyalamayı amaçlıyor, o yüzden bu verimsizlikleri hafifletebilecek bir yola sahip değil. şu anda, ethereum bloklarının kanıtlarının üretilmesi saatler sürüyor. bu durum, kanıtlayıcıyı büyük ölçüde paralelleştirmeye yönelik zekice mühendislikle veya uzun vadede zk-snark asic'ler ile hafifletilebilir. bunu kim inşa ediyor? zk-evm topluluk sürümü (privacy and scaling explorations, scroll ekibi, taiko ve diğerlerinin topluluk katkılarıyla başlatılan) tier 1 bir zk-evm'dir. tip 2 (tamamen evm-dengi) tip 2 zk-evm'ler tam anlamıyla evm-dengi olmaya çalışır, ancak tam olarak ethereum-dengi değildirler. yani, "içeriden" bakıldığında tam olarak ethereum gibi görünürler, ancak dışarıda bazı farklılıklar, özellikle blok yapısı ve state tree gibi veri yapılarında bulunur. hedef, mevcut uygulamalarla tamamen uyumlu olmak, ancak geliştirmeyi kolaylaştırmak ve kanıt oluşturmayı daha hızlı hale getirmek için ethereum'da bazı küçük değişiklikler yapmaktır. avantaj: vm seviyesinde mükemmel denklik tip 2 zk-evm'ler, ethereum durumu gibi verileri tutan veri yapılarına değişiklikler yaparlar. neyse ki, bunlar evm'in doğrudan erişemeyeceği yapılar olduğundan, ethereum üzerinde çalışan uygulamalar neredeyse her zaman tip 2 zk-evm rollup'larında da çalışmaya devam eder. ethereum execution istemcilerini olduğu gibi kullanamazsınız, ancak bazı değişikliklerle kullanabilirsiniz ve hala evm hata ayıklama araçlarını ve çoğu diğer geliştirici altyapısını kullanabilirsiniz. az sayıda istisna vardır. örneğin geçmiş işlemler, makbuzlar veya durum hakkındaki iddiaları doğrulamak için geçmiş ethereum bloklarının merkle kanıtlarını doğrulayan uygulamalarda bir uyumsuzluk ortaya çıkar (örn. köprüler bazen bunu yapar). keccak'ı farklı bir hash fonksiyonuyla değiştiren bir zk-evm, bu kanıtları bozabilir. bununla birlikte, zaten genellikle uygulamaların bu şekilde inşa edilmesini önermem, çünkü gelecek ethereum değişiklikleri (örn. verkle ağaçları) bu tür uygulamaları ethereum'un kendisinin üzerinde bile bozacaktır. daha iyi bir alternatif, ethereum'un geleceğe-hazır geçmiş erişim ön-derlemelerini eklemesidir. dezavantaj: gelişmiş ancak hala yavaş kanıtlayıcı süresi tip 2 zk-evm'ler, ethereum yığınının gereksiz derecede karmaşık ve zk dostu olmayan kriptografiye dayanan kısımlarını kaldırarak tip 1'den daha hızlı kanıtlayıcı süreleri sağlar. özellikle ethereum'un keccak'ını ve rlp tabanlı merkle patricia ağacını ve belki de blok ve makbuz yapılarını değiştirebilirler. tip 2 zk-evm'ler, farklı bir hash fonksiyonu kullanabilirler, örneğin poseidon. başka bir doğal değişiklik, state tree'yi kod hash'ini ve keccak'ı depolayacak şekilde değiştirerek extcodehash ve extcodecopy opcode'larını işlemek için hash doğrulama ihtiyacını ortadan kaldırmaktır. bu değişiklikler kanıtlayıcı sürelerini önemli ölçüde iyileştirir ancak her sorunu çözmez. evm'i olduğu gibi kanıtlama zorunluluğundan kaynaklanan yavaşlık, evm'in doğasında bulunan tüm verimsizlikler ve zk dostu olmama durumuyla birlikte hala devam ediyor. bunun basit bir örneği bellektir: bir mload, "hizalanmamış" parçalar (başlangıç ve bitişin 32'nin katı olmadığı) dahil olmak üzere herhangi bir 32 byte'ı okuyabildiğinden, bir mload basitçe bir parçayı okuyormuş gibi yorumlanamaz; bunun yerine, sonucu birleştirmek için iki ardışık parçanın okunmasını ve bit işlemleri gerçekleştirilmesini gerektirebilir. bunu kim inşa ediyor? scroll'un zk-evm projesi, polygon hermez gibi tip 2 zk-evm'e doğru ilerliyor. bununla birlikte, her iki proje de henüz tam olarak oraya ulaşmadı; özellikle daha karmaşık ön-derlemelerin çoğu henüz implement edilmedi. dolayısıyla şu anda her iki projenin de tip 3. olarak değerlendirilmesi daha iyi. tip 2.5 (evm-dengi, gaz ücretleri hariç) en kötü durumda kanıt sürelerini önemli ölçüde iyileştirmenin bir yolu, evm'deki zk-kanıtlaması çok zor olan belirli işlemlerin gaz ücretlerini büyük ölçüde artırmaktır. bu, ön-derlemeleri, keccak opcode'unu ve muhtemelen belirli kontrat çağırma kalıplarını veya belleğe/depolamaya erişmeyi veya geri döndürmeyi (revert) içerebilir. gaz ücretlerini değiştirmek, geliştirici araçlarının uyumluluğunu azaltabilir ve birkaç uygulamayı bozabilir, ancak genellikle "derin" evm değişikliklerinden daha az riskli olarak kabul edilir. geliştiriciler, bir işlemde bir bloğa sığmayacak kadar fazla gaz gerektirmemeye ve hard-coded gaz miktarlarıyla çağrılar yapmamaya dikkat etmelidir (bu zaten geliştiriciler için uzun süredir standart bir öneridir). kaynak sınırlamalarını yönetmenin alternatif bir yolu, her operasyonun kaç kez çağrılabileceği konusunda katı sınırlar belirlemektir. bunun devrelerde uygulanması daha kolaydır, ancak evm güvenlik varsayımlarıyla daha az uyumludur. ben buna tip 2.5 yerine tip 3 yaklaşımı derim. tip 3 (neredeyse evm-dengi) tip 3 zk-evm'ler neredeyse evm-dengidir, ancak kanıt sürelerini daha da iyileştirmek ve evm'i geliştirmeyi kolaylaştırmak için tam denklikten ödün verirler. avantaj: daha kolay inşa edilir ve daha hızlı kanıtlayıcı süreleri sunar. tip 3 zk-evm'ler, zk-evm implementasyonunda uygulanması son derece zor olan birkaç özelliği kaldırabilir. genellikle ön-derlemeler bu listede en üsttedir. ek olarak, tip 3 zk-evm'lerin bazen kontrat kodunu, belleği veya yığını nasıl ele aldıkları konusunda da küçük farklılıkları vardır. dezavantaj: daha fazla uyumsuzluk. tip 3 zk-evm'in hedefi, çoğu uygulama ile uyumlu olup geri kalanları için sadece minimal yeniden yazım gerektirmektir. bununla beraber, tip 3 zk-evm'in kaldırdığı ön-derlemeleri kullanan veya vm'lerin farklı işlediği ince ayrıntılardaki bağımlılıklardan dolayı yeniden yazılması gereken bazı uygulamalar olacaktır. bunu kim inşa ediyor? scroll ve polygon'un her ikisi de mevcut formlarıyla tip 3'tür, ancak zamanla uyumluluğu artırmaları bekleniyor. polygon, zkasm denen kendi dahili dilini zk-doğruladıkları ve zkasm implementasyonunu kullanarak zk-evm kodunu yorumladıkları benzersiz bir tasarıma sahiptir. bu implementasyon detayına rağmen yine de buna hakiki tip 3 zk-evm diyeceğim; çünkü hala evm kodunu doğrulayabiliyor, sadece bunu yapmak için yalnızca farklı bir dahili mantık kullanır. bugün, hiçbir zk-evm ekibi tip 3 olmak istemez; tip 3, ön-derlemelerin eklenmesi gibi karmaşık işler bitene ve proje tip 2.5'a geçinceye kadar sadece bir geçiş aşamasıdır. ancak gelecekte tip 1 veya tip 2 zk-evm'ler, düşük kanıtlayıcı süreleri ve gaz ücretleri ile geliştiriciler için işlevsellik sağlayan yeni zk-snark dostu ön-derlemeler ekleyerek gönüllü olarak tip 3 zk-evm'lere dönüşebilirler. tip 4 (yüksek-seviye-dil denkliği) bir tip 4 sistemi, yüksek seviyeli bir dilde (örn. solidity, vyper veya her ikisinin de derlediği bir ara dilde) yazılmış akıllı kontrat kaynak kodunu alarak ve bunu açıkça zk-snark dostu olacak şekilde tasarlanmış bir dilde derleyerek çalışır. avantaj: çok hızlı kanıtlayıcı süreleri her evm yürütülme işleminin tüm farklı bölümlerini zk-kanıtlamayarak ve doğrudan yüksek seviye koddan başlayarak önlenebilecek çok sayıda ek yük vardır. bu avantajı bu yazımda sadece bir cümleyle açıklıyorum (aşağıdaki, uyumlulukla ilgili dezavantajlar için olan büyük listeyle karşılaştırıldığında), ancak bu bir değer yargısı olarak yorumlanmamalıdır! yüksek seviyeli dillerden doğrudan derlemek gerçekten maliyetleri büyük ölçüde azaltabilir ve bir kanıtlayıcı olmayı daha kolay hale getirerek merkeziyetsizleşmeye yardımcı olabilir. dezavantaj: daha fazla uyumsuzluk vyper veya solidity ile yazılmış "normal" bir uygulama derlenebilir ve "sadece çalışır", ancak pek çok uygulamanın "normal" olmadığı bazı önemli yollar vardır: kontratlar tip 4 bir sistemde evm'de olduğu gibi aynı adreslere sahip olmayabilir, çünkü create2 kontrat adresleri tamı tamına bytecode'a bağlıdır. bu, henüz deploy edilmemiş "karşıolgusal kontratlara", erc-4337 cüzdanlara, eip-2470 tekillerine ve birçok diğer uygulamaya dayanan uygulamaları bozar. elle yazılmış evm bytecode kullanmak daha zordur. birçok uygulama, verimlilik için bazı bölümlerinde elle yazılmış evm bytecode kullanır. tip 4 sistemleri bunu desteklemeyebilir, ancak bu kullanım durumlarını karşılamak için tam anlamıyla bir tip 3 zk-evm olmanın çabasına girmeden sınırlı evm bytecode desteği uygulamanın yolları vardır. birçok hata ayıklama altyapısı taşınamaz, çünkü bu tür altyapılar evm bytecode'u üzerinden çalışır. bununla birlikte, bu dezavantaj "geleneksel" yüksek seviyeli veya ara diller (örn. llvm) tarafından sunulan hata ayıklama altyapısına daha fazla erişim ile hafifletilir. geliştiriciler bu konuları akıllarında bulundurmalıdır. bunu kim inşa ediyor? zksync bir tip 4 sistemidir ancak zamanla evm bytecode'u için uyumluluk ekleyebilir. nethermind'ın warp projesi, solidity'den starkware'in cairo'suna derleyici inşa ediyor ve bu, starknet'i fiili bir tip 4 sistemine dönüştürecektir. the future of zk-evm types tipler diğer tiplerden açıkça "daha iyi" veya "daha kötü" değildir. aksine, bunlar takas alanındaki farklı noktalardır: daha düşük-numaralı tipler mevcut altyapı ile daha uyumludur, ancak daha yavaştır ve daha yüksek-numaralı tipler mevcut altyapı ile daha az uyumludur, ancak daha hızlıdır. genel olarak bu tiplerin tamamının keşfedilmesi alan açısından sağlıklıdır. ek olarak, zk-evm projeleri kolaylıkla yüksek-numaralı tiplerden başlayabilir ve zamanla daha düşük-numaralı tiplere atlayabilir (veya tam tersi). örneğin: bir zk-evm, özellikle zk-kanıtlanması zor olan bazı özellikleri dahil etmemeye karar vererek tip 3 olarak başlayabilir. daha sonra zamanla bu özellikleri ekleyip tip 2'ye geçebilirler. bir zk-evm, tip 2 olarak başlayabilir ve daha sonra, tamamen ethereum'a uyumluluk modu ile veya daha hızlı kanıtlanabilen değiştirilmiş bir state tree ile çalışma olanağı sunarak hibrit bir tip 2 / tip 1 zk-evm haline gelebilir. scroll bu yönde ilerlemeyi düşünüyor. tip 4 olarak başlayan sistem, daha sonra evm kodunu işleme yeteneğinin eklenmesiyle zamanla tip 3 haline gelebilir (yine de geliştiriciler ücretleri ve kanıt sürelerini azaltmak için yüksek seviyeli dillerle doğrudan derlemeye yönelecektir). tip 2 veya tip 3 zk-evm, ethereum'un daha zk dostu olma çabasıyla modifikasyonlarını benimsemesi durumunda tip 1 zk-evm haline gelebilir. bir tip 1 veya tip 2 zk-evm, oldukça zk-snark dostu bir dilde kodu doğrulamak için bir ön-derleme ekleyerek tip 3 zk-evm haline gelebilir. bu, geliştiricilere ethereum uyumluluğu ve hız arasında bir seçenek sunar. bu tip 3 olacaktır çünkü mükemmel evm-denkliğini bozar, ancak pratik amaçlar ve niyetler için tip 1 ve 2'nin birçok avantajına sahip olur. ana dezavantaj, bazı geliştirici araçlarının zk-evm'in özel ön-derlemelerini anlamayabileceğidir, ancak bu sorun düzeltilebilir: geliştirici araçları, ön-derlemenin evm kodu dengi bir implementasyonunu içeren bir yapılandırma formatını destekleyerek evrensel ön-derleme desteği ekleyebilir. şahsen, benim umudum zk-evm'lerdeki iyileştirmeler ve ethereum'un kendisini daha zk-snark dostu hale getirecek iyileştirmelerin bir araya gelmesiyle her şeyin zamanla tip 1 haline gelmesidir. böyle bir gelecekte, hem zk rollup'ları hem de ethereum zincirinin kendisini doğrulamak için kullanılabilecek birden fazla zk-evm implementasyonuna sahip olurduk. teorik olarak ethereum'un l1 kullanımı için tek bir zk-evm implementasyonu üzerinde standartlaşmasına gerek yoktur; farklı istemciler farklı kanıtlar kullanabilir, bu nedenle kod fazlalığından yararlanmaya devam edelim. ancak böyle bir geleceğe ulaşmamız oldukça zaman alacak. bu esnada, ethereum ve ethereum tabanlı zk-rollup'larını ölçeklendirmenin farklı yollarında birçok inovasyon göreceğiz. the subjectivity / exploitability tradeoff | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the subjectivity / exploitability tradeoff posted by vitalik buterin on february 14, 2015 research & development one of the issues inherent in many kinds of consensus architectures is that although they can be made to be robust against attackers or collusions up to a certain size, if an attacker gets large enough they are still, fundamentally, exploitable. if attackers in a proof of work system have less than 25% of mining power and everyone else is non-colluding and rational, then we can show that proof of work is secure; however, if an attacker is large enough that they can actually succeed, then the attack costs nothing and other miners actually have the incentive to go along with the attack. schellingcoin, as we saw, is vulnerable to a so-called p + epsilon attack in the presence of an attacker willing to commit to bribing a large enough amount, and is itself capturable by a majority-controlling attacker in much the same style as proof of work. one question that we may want to ask is, can we do better than this? particularly if a pseudonymous cryptocurrency like bitcoin succeeds, and arguably even if it does not, there doubtlessly exists some shadowy venture capital industry willing to put up the billions of dollars needed to launch such attacks if they can be sure that they can quickly earn a profit from executing them. hence, what we would like to have is cryptoeconomic mechanisms that are not just stable, in the sense that there is a large margin of minimum "size" that an attacker needs to have, but also unexploitable although we can never measure and account for all of the extrinsic ways that one can profit from attacking a protocol, we want to at the very least be sure that the protocol presents no intrinsic profit potential from an attack, and ideally a maximally high intrinsic cost. for some kinds of protocols, there is such a possibility; for example, with proof of stake we can punish double-signing, and even if a hostile fork succeeds the participants in the fork would still lose their deposits (note that to properly accomplish this we need to add an explicit rule that forks that refuse to include evidence of double-signing for some time are to be considered invalid). unfortunately, for schellingcoin-style mechanisms as they currently are, there is no such possibility. there is no way to cryptographically tell the difference between a schellingcoin instance that votes for the temperature in san francisco being 4000000000'c because it actually is that hot, and an instance that votes for such a temperature because the attacker committed to bribe people to vote that way. voting-based daos, lacking an equivalent of shareholder regulation, are vulnerable to attacks where 51% of participants collude to take all of the dao's assets for themselves. so what can we do? between truth and lies one of the key properties that all of these mechanisms have is that they can be described as being objective: the protocol's operation and consensus can be maintained at all times using solely nodes knowing nothing but the full set of data that has been published and the rules of the protocol itself. there is no additional "external information" (eg. recent block hashes from block explorers, details about specific forking events, knowledge of external facts, reputation, etc) that is required in order to deal with the protocol securely. this is in contrast to what we will describe as subjective mechanisms mechanisms where external information is required to securely interact with them. when there exist multiple levels of the cryptoeconomic application stack, each level can be objective or subjective separately: codius allows for subjectively determined scoring of oracles for smart contract validation on top of objective blockchains (as each individual user must decide for themselves whether or not a particular oracle is trustworthy), and ripple's decentralized exchange provides objective execution on top of an ultimately subjective blockchain. in general, however, cryptoeconomic protocols so far tend to try to be objective where possible. objectivity has often been hailed as one of the primary features of bitcoin, and indeed it has many benefits. however, at the same time it is also a curse. the fundamental problem is this: as soon as you try to introduce something extra-cryptoeconomic, whether real-world currency prices, temperatures, events, reputation, or even time, from the outside world into the cryptoeconomic world, you are trying to create a link where before there was absolutely none. to see how this is an issue, consider the following two scenarios: the truth is b, and most participants are honestly following the standard protocol through which the contract discovers that the truth is b, but 20% are attackers or accepted a bribe. the truth is a, but 80% of participants are attackers or accepted a bribe to pretend that the truth is b. from the point of view of the protocol, the two are completely indistinguishable; between truth and lies, the protocol is precisely symmetrical. hence, epistemic takeovers (the attacker convincing everyone else that they have convinced everyone else to go along with an attack, potentially flipping an equilibrium at zero cost), p + epsilon attacks, profitable 51% attacks from extremely wealthy actors, etc, all begin to enter the picture. although one might think at first glance that objective systems, with no reliance on any actor using anything but information supplied through the protocol, are easy to analyze, this panoply of issues reveals that to a large extent the exact opposite is the case: objective protocols are vulnerable to takeovers, and potentially zero-cost takeovers, and standard economics and game theory quite simply have very bad tools for analyzing equilibrium flips. the closest thing that we currently have to a science that actually does try to analyze the hardness of equilibrium flips is chaos theory, and it will be an interesting day when crypto-protocols start to become advertised as "chaos-theoretically guaranteed to protect your grandma's funds". hence, subjectivity. the power behind subjectivity lies in the fact that concepts like manipulation, takeovers and deceit, not detectable or in some cases even definable in pure cryptography, can be understood by the human community surrounding the protocol just fine. to see how subjectivity may work in action, let us jump straight to an example. the example supplied here will define a new, third, hypothetical form of blockchain or dao governance, which can be used to complement futarchy and democracy: subjectivocracy. pure subjectivocracy is defined quite simply: if everyone agrees, go with the unanimous decision. if there is a disagreement, say between decision a and decision b, split the blockchain/dao into two forks, where one fork implements decision a and the other implements decision b. all forks are allowed to exist; it's left up to the surrounding community to decide which forks they care about. subjectivocracy is in some sense the ultimate non-coercive form of governance; no one is ever forced to accept a situation where they don't get their own way, the only catch being that if you have policy preferences that are unpopular then you will end up on a fork where few others are left to interact with you. perhaps, in some futuristic society where nearly all resources are digital and everything that is material and useful is too-cheap-to-meter, subjectivocracy may become the preferred form of government; but until then the cryptoeconomy seems like a perfect initial use case. for another example, we can also see how to apply subjectivocracy to schellingcoin. first, let us define our "objective" version of schellingcoin for comparison's sake: the schellingcoin mechanism has an associated sub-currency. anyone has the ability to "join" the mechanism by purchasing units of the currency and placing them as a security deposit. weight of participation is proportional to the size of the deposit, as usual. anyone has the ability to ask the mechanism a question by paying a fixed fee in that mechanism's currency. for a given question, all voters in the mechanism vote either a or b. everyone who voted with the majority gets a share of the question fee; everyone who voted against the majority gets nothing. note that, as mentioned in the post on p + epsilon attacks, there is a refinement by paul sztorc under which minority voters lose some of their coins, and the more "contentious" a question becomes the more coins minority voters lose, right up to the point where at a 51/49 split the minority voters lose all their coins to the majority. this substantially raises the bar for a p + epsilon attack. however, raising the bar for us is not quite good enough; here, we are interested in having no exploitability (once again, we formally define "exploitability" as "the protocol provides intrinsic opportunities for profitable attacks") at all. so, let us see how subjectivity can help. we will elide unchanged details: for a given question, all voters in the mechanism vote either a or b. if everyone agrees, go with the unanimous decision and reward everyone. if there is a disagreement, split the mechanism into two on-chain forks, where one fork acts as if it chose a, rewarding everyone who voted a, and the other fork acts as if it chose b, rewarding everyone who voted b. each copy of the mechanism has its own sub-currency, and can be interacted with separately. it is up to the user to decide which one is more worth asking questions to. the theory is that if a split does occur, the fork specifying the correct answer will have increased stake belonging to truth-tellers, the fork specifying the wrong answer will have increased stake belonging to liars, and so users will prefer to ask questions to the fork where truth-tellers have greater influence. if you look at this closely, you can see that this is really just a clever formalism for a reputation system. all that the system does is essentially record the votes of all participants, allowing each individual user wishing to ask a question to look at the history of each respondent and then from there choose which group of participants to ask. a very mundane, old-fashioned, and seemingly really not even all that cryptoeconomic approach to solving the problem. now, where do we go from here? moving to practicality pure subjectivocracy, as described above, has two large problems. first, in most practical cases, there are simply far too many decisions to make in order for it to be practical for users to decide which fork they want to be on for every single one. in order to prevent massive cognitive load and storage bloat, it is crucial for the set of subjectively-decided decisions to be as small as possible. second, if a particular user does not have a strong belief that a particular decision should be answered in one way or another (or, alternatively, does not know what the correct decision is), then that user will have a hard time figuring out which fork to follow. this issue is particularly strong in the context of a category that can be termed "very stupid users" (vsus) think not homer simpson, but homer simpson's fridge. examples include internet-of-things/smart property applications (eg. suvs), other cryptoeconomic mechanisms (eg. ethereum contracts, separate blockchains, etc), hardware devices controlled by daos, independently operating autonomous agents, etc. in short, machines that have (i) no ability to get updated social information, and (ii) no intelligence beyond the ability to follow a pre-specified protocol. vsus exist, and it would be nice to have some way of dealing with them. the first problem, surprisingly enough, is essentially isomorphic to another problem that we all know very well: the blockchain scalability problem. the challenge is exactly the same: we want to have the strength equivalent to all users performing a certain kind of validation on a system, but not require that level of effort to actually be performed every time. and in blockchain scalability we have a known solution: try to use weaker approaches, like randomly selected consensus groups, to solve problems by default, only using full validation as a fallback to be used if an alarm has been raised. here, we will do a similar thing: try to use traditional governance to resolve relatively non-contentious issues, only using subjectivocracy as a sort of fallback and incentivizer-of-last-resort. so, let us define yet another version of schellingcoin: for a given question, all voters in the mechanism vote either a or b. everyone who voted with the majority gets a share of the question fee (which we will call p); everyone who voted against the majority gets nothing. however, deposits are frozen for one hour after voting ends. a user has the ability to put down a very large deposit (say, 50*p) to "raise the alarm" on a particular question that was already voted on essentially, a bet saying "this was done wrong". if this happens, then the mechanism splits into two on-chain forks, with one answer chosen on one fork and the other answer chosen on the other fork. on the fork where the chosen answer is equal to the original voted answer, the alarm raiser loses the deposit. on the other form, the alarm raiser gets back a reward of 2x the deposit, paid out from incorrect voters' deposits. additionally, the rewards for all other answerers are made more extreme: "correct" answerers get 5*p and "incorrect" answerers lose 10*p. if we make a maximally generous assumption and assume that, in the event of a split, the incorrect fork quickly falls away and becomes ignored, the (partial) payoff matrix starts to look like this (assuming truth is a): you vote a you vote b you vote against consensus, raise the alarm others mainly vote a p 0 -50p 10p = -60p others mainly vote a, n >= 1 others raise alarm 5p -10p -10p (50 / (n + 1)) * p others mainly vote b 0 p 50p + 5p = 55p others mainly vote b, n >= 1 others raise alarm 5p -10p 5p + (50 / (n + 1)) * p the strategy of voting with the consensus and raising the alarm is clearly self-contradictory and silly, so we will omit it for brevity. we can analyze the payoff matrix using a fairly standard repeated-elimination approach: if others mainly vote b, then the greatest incentive is for you to raise the alarm. if others mainly vote a, then the greatest incentive is for you to vote a. hence, each individual will never vote b. hence, we know that everyone will vote a, and so everyone's incentive is to vote a. note that, unlike the schellingcoin game, there is actually a unique equilibrium here, at least if we assume that subjective resolution works correctly. hence, by relying on what is essentially game theory on the part of the users instead of the voters, we have managed to avoid the rather nasty set of complications involving multi-equilibrium games and instead have a clearer analysis. additionally note that the "raise the alarm by making a bet" protocol differs from other approaches to fallback protocols that have been mentioned in previous articles here in the context of scalability; this new mechanism is superior to and cleaner than those other approaches, and can be applied in scalability theory too. the public function of markets now, let us bring our cars, blockchains and autonomous agents back into the fold. the reason why bitcoin's objectivity is so valued is to some extent precisely because the objectivity makes it highly amenable to such applications. thus, if we want to have a protocol that competes in this regard, we need to have a solution for these "very stupid users" among us as well. enter markets. the key insight behind hayek's particular brand of libertarianism in the 1940s, and robin hanson's invention of futarchy half a century later, is the idea that markets exist not just to match buyers and sellers, but also to provide a public service of information. a prediction market on a datum (eg. gdp, unemployment, etc) reveals the information of what the market thinks will be value of that datum at some point in the future, and a market on a good or service or token reveals to interested individuals, policymakers and mechanism designers how much the public values that particular good or service or token. thus, markets can be thought of as a complement to schellingcoin in that they, like schellingcoin, are also a window between the digital world and the "real" world in this case, a window that reveals just how much the real world cares about something. so, how does this secondary "public function" of markets apply here? in short, the answer is quite simple. suppose that there exists a schellingcoin mechanism, of the last type, and after one particular question two forks appear. one fork says that the temperature in san francisco is 20'c; the other fork says that the temperature is 4000000000'c. as a vsu, what do you see? well, let's see what the market sees. on the one hand, you have a fork where the larger share of the internal currency is controlled by truth-tellers. on the other hand, you have a fork where the larger share is controlled by liars. well, guess which of the two currencies has a higher price on the market... in cryptoeconomic terms, what happened here? simply put, the market translated the human intelligence of the intelligent users in what is an ultimately subjective protocol into a pseudo-objective signal that allows the vsus to join onto the correct fork as well. note that the protocol itself is not objective; even if the attacker manages to successfully manipulate the market for a brief period of time and massively raise the price of token b, the users are still going to have a higher valuation for token a, and when the manipulator gives up token a will go right back to being the dominant one. now, what are the robustness properties of this market against attack? as was brought up in the hanson/moldbug debate on futarchy, in the ideal case a market will provide the correct price for a token for as long as the economic weight of the set of honestly participating users exceeds the economic weight of any particular colluding set of attackers. if some attackers bid the price up, an incentive arises for other participants to sell their tokens and for outsiders to come in and short it, in both cases earning an expected profit and at the same time helping to push the price right back down to the correct value. in practice, manipulation pressure does have some effect, but a complete takeover is only possible if the manipulator can outbid everyone else combined. and even if the attacker does succeed, they pay dearly for it, buying up tokens that end up being nearly valueless once the attack ends and the fork with the correct answer reasserts itself as the most valuable fork on the market. of course, the above is only a sketch of how quasi-subjective schellingcoin may work; in reality a number of refinements will be needed to disincentivize asking ambiguous or unethical questions, handling linear and not just binary bets, and optimizing the non-exploitability property. however, if p + epsilon attacks, profit-seeking 51% attacks, or any other kind of attack ever actually do become a problem with objective schellingcoin mechanisms, the basic model stands ready as a substitute. listening to markets and proof of work earlier in this post, and in my original post on schellingcoin, i posited a sort of isomorphism between schellingcoin and proof of work in the original post reasoning that because proof of work works so will schellingcoin, and above that because schellingcoin is problematic so is proof of work. here, let us expand on this isomorphism further in a third direction: if schellingcoin can be saved through subjectivity, then perhaps so can proof of work. the key argument is this: proof of work, at the core, can be seen in two different ways. one way of seeing proof of work is as a schellingcoin contest, an objective protocol where the participants that vote with the majority get rewarded 25 btc and everyone else gets nothing. the other approach, however, is to see proof of work as a sort of constant ongoing "market" between a token and a resource that can be measured purely objectively: computational power. proof of work is an infinite opportunity to trade computational power for currency, and the more interest there is in acquiring units in a currency the more work will be done on its blockchain. "listening" to this market consists simply of verifying and computing the total quantity of work. seeing the description in the previous section of how our updated version of schellingcoin might work, you may have been inclined to propose a similar approach for cryptocurrency, where if a cryptocurrency gets forked one can see the price of both forks on an exchange, and if the exchange prices one fork much more highly that implies that that fork is legitimate. however, such an approach has a problem: determining the validity of a crypto-fiat exchange is subjective, and so the problem is beyond the reach of a vsu. but with proof of work as our "exchange", we can actually get much further. here is the equivalence: exponential subjective scoring. in ess, the "score" that a client attaches to a fork depends not just on the total work done on the fork, but also on the time at which the fork appeared; forks that come later are explicitly penalized. hence, the set of always-online users can see that a given fork came later, and therefore that it is a hostile attack, and so they will refuse to mine on it even if its proof of work chain grows to have much more total work done on it. their incentive to do this is simple: they expect that eventually the attacker will give up, and so they will continue mining and eventually overtake the attacker, making their fork the universally accepted longest one again; hence, mining on the original fork has an expected value of 25 btc and mining on the attacking fork has an expected value of zero. vsus that are not online at the time of a fork will simply look at the total proof of work done; this strategy is equivalent to the "listen to the child with the higher price" approach in our version of schellingcoin. during an attack, such vsus may of course temporarily be tricked, but eventually the original fork will win and so the attacker will have massively paid for the treachery. hence, the subjectivity once again makes the mechanism less exploitable. conclusion altogether, what we see is that subjectivity, far from being an enemy of rigorous analysis, in fact makes many kinds of game-theoretic analysis of cryptoeconomic protocols substantially easier. however, if this kind of subjective algorithm design becomes accepted as the most secure approach, it has far-reaching consequences. first of all, bitcoin maximalism, or any kind of single-cryptocurrency maximalism generally, cannot survive. subjective algorithm design inherently requires a kind of loose coupling, where the higher-level mechanism does not actually control anything of value belonging to a lower-level protocol; this condition is necessary in order to allow higher-level mechanism instances to copy themselves. in fact, in order for the vsu protocol to work, every mechanism would need to contain its own currency which would rise and fall with its perceived utility, and so thousands or even millions of "coins" would need to exist. on the other hand, it may well be possible to enumerate a very specific number of mechanisms that actually need to be subjective perhaps, basic consensus on block data availability validation and timestamping and consensus on facts, and everything else can be built objectively on top. as is often the case, we have not even begun to see substantial actual attacks take place, and so it may well be over a decade until anything close to a final judgement needs to be made. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle what do i think about network states? 2022 jul 13 see all posts on july 4, balaji srinivasan released the first version of his long-awaited new book describing his vision for "network states": communities organized around a particular vision of how to run their own society that start off as online clubs, but then build up more and more of a presence over time and eventually become large enough to seek political autonomy or even diplomatic recognition. network states can be viewed as an attempt at an ideological successor to libertarianism: balaji repeatedly praises the sovereign individual (see my mini-review here) as important reading and inspiration, but also departs from its thinking in key ways, centering in his new work many non-individualistic and non-monetary aspects of social relations like morals and community. network states can also be viewed as an attempt to sketch out a possible broader political narrative for the crypto space. rather than staying in their own corner of the internet disconnected from the wider world, blockchains could serve as a centerpiece for a new way of organizing large chunks of human society. these are high promises. can network states live up to them? do network states actually provide enough benefits to be worth getting excited about? regardless of the merits of network states, does it actually make sense to tie the idea together with blockchains and cryptocurrency? and on the other hand, is there anything crucially important that this vision of the world misses? this post represents my attempt to try to understand these questions. table of contents what is a network state? so what kinds of network states could we build? what is balaji's megapolitical case for network states? do you have to like balaji's megapolitics to like network states? what does cryptocurrency have to do with network states? what aspects of balaji's vision do i like? what aspects of balaji's vision do i take issue with? non-balajian network states is there a middle way? what is a network state? balaji helpfully gives multiple short definitions of what a network state is. first, his definition in one sentence: a network state is a highly aligned online community with a capacity for collective action that crowdfunds territory around the world and eventually gains diplomatic recognition from pre-existing states. this so far seems uncontroversial. create a new internet community online, once it grows big enough materialize it offline, and eventually try to negotiate for some kind of status. someone of almost any political ideology could find some form of network state under this definition that they could get behind. but now, we get to his definition in a longer sentence: a network state is a social network with a moral innovation, a sense of national consciousness, a recognized founder, a capacity for collective action, an in-person level of civility, an integrated cryptocurrency, a consensual government limited by a social smart contract, an archipelago of crowdfunded physical territories, a virtual capital, and an on-chain census that proves a large enough population, income, and real-estate footprint to attain a measure of diplomatic recognition. here, the concept starts to get opinionated: we're not just talking about the general concept of online communities that have collective agency and eventually try to materialize on land, we're talking about a specific balajian vision of what network states should look like. it's completely possible to support network states in general, but have disagreements with the balajian view of what properties network states should have. if you're not already a "crypto convert", it's hard to see why an "integrated cryptocurrency" is such a fundamental part of the network state concept, for example though balaji does later on in the book defend his choices. finally, balaji expands on this conception of a balajian network state in longer-form, first in "a thousand words" (apparently, balajian network states use base 8, as the actual word count is exactly \(512 = 8^3\)) and then an essay, and at the very end of the book a whole chapter. and, of course, an image. one key point that balaji stresses across many chapters and pages is the unavoidable moral ingredient required for any successful new community. as balaji writes: the quick answer comes from paul johnson at the 11:00 mark of this talk, where he notes that early america's religious colonies succeeded at a higher rate than its for-profit colonies, because the former had a purpose. the slightly longer answer is that in a startup society, you're not asking people to buy a product (which is an economic, individualistic pitch) but to join a community (which is a cultural, collective pitch). the commitment paradox of religious communes is key here: counterintuitively, it's the religious communes that demand the most of their members that are the most long-lasting. this is where balajism explicitly diverges from the more traditional neoliberal-capitalist ideal of the defanged, apolitical and passion-free consumerist "last man". unlike the strawman libertarian, balaji does not believe that everything can "merely be a consumer product". rather, he stresses greatly the importance of social norms for cohesion, and a literally religious attachment to the values that make a particular network state distinct from the world outside. as balaji says in this podcast at 18:20, most current libertarian attempts at micronations are like "zionism without judaism", and this is a key part of why they fail. this recognition is not a new one. indeed, it's at the core of antonio garcia martinez's criticism of balaji's earlier sovereign-individual ideas (see this podcast at ~27:00), praising the tenacity of cuban exiles in miami who "perhaps irrationally, said this is our new homeland, this is our last stand". and in fukuyama's the end of history: this city, like any city, has foreign enemies and needs to be defended from outside attack. it therefore needs a class of guardians who are courageous and public-spirited, who are willing to sacrifice their material desires and wants for the sake of the common good. socrates does not believe that courage and public-spiritedness can arise out of a calculation of enlightened self-interest. rather, they must be rooted in thymos, in the just pride of the guardian class in themselves and in their own city, and their potentially irrational anger against those who threaten it. balaji's argument in the network state, as i am interpreting it, is as follows. while we do need political collectives bound not just by economic interest but also by moral force, we don't need to stick with the specific political collectives we have today, which are highly flawed and increasingly unrepresentative of people's values. rather, we can, and should, create new and better collectives and his seven-step program tells us how. so what kinds of network states could we build? balaji outlines a few ideas for network states, which i will condense into two key directions: lifestyle immersion and pro-tech regulatory innovation. balaji's go-to example for lifestyle immersion is a network state organized around health: next, let's do an example which requires a network archipelago (with a physical footprint) but not a full network state (with diplomatic recognition). this is keto kosher, the sugar-free society. start with a history of the horrible usda food pyramid, the grain-heavy monstrosity that gave cover to the corporate sugarification of the globe and the obesity epidemic. ... organize a community online that crowdfunds properties around the world, like apartment buildings and gyms, and perhaps eventually even culdesacs and small towns. you might take an extreme sugar teeotaler approach, literally banning processed foods and sugar at the border, thereby implementing a kind of "keto kosher". you can imagine variants of this startup society that are like "carnivory communities" or "paleo people". these would be competing startup societies in the same broad area, iterations on a theme. if successful, such a society might not stop at sugar. it could get into setting cultural defaults for fitness and exercise. or perhaps it could bulk purchase continuous glucose meters for all members, or orders of metformin. this, strictly speaking, does not require any diplomatic recognition or even political autonomy though perhaps, in the longer-term future, such enclaves could negotiate for lower health insurance fees and medicare taxes for their members. what does require autonomy? well, how about a free zone for medical innovation? now let's do a more difficult example, which will require a full network state with diplomatic recognition. this is the medical sovereignty zone, the fda-free society. you begin your startup society with henninger's history of fda-caused drug lag and tabarrok's history of fda interference with so-called "off label" prescription. you point out how many millions were killed by its policies, hand out t-shirts like act-up did, show dallas buyers club to all prospective residents, and make clear to all new members why your cause of medical sovereignty is righteous ... for the case of doing it outside the us, your startup society would ride behind, say, the support of the malta's fda for a new biomedical regime. for the case of doing it within the us, you'd need a governor who'd declare a sanctuary state for biomedicine. that is, just like a sanctuary city declares that it won't enforce federal immigration law, a sanctuary state for biomedicine would not enforce fda writ. one can think up of many more examples for both categories. one could have a zone where it's okay to walk around naked, both securing your legal right to do so and helping you feel comfortable by creating an environment where many other people are naked too. alternatively, you could have a zone where everyone can only wear basic plain-colored clothing, to discourage what's perceived as a zero-sum status competition of expending huge effort to look better than everyone else. one could have an intentional community zone for cryptocurrency users, requiring every store to accept it and demanding an nft to get in the zone at all. or one could build an enclave that legalizes radical experiments in transit and drone delivery, accepting higher risks to personal safety in exchange for the privilege of participating in a technological frontier that will hopefully set examples for the world as a whole. what is common about all of these examples is the value of having a physical region, at least of a few hectares, where the network state's unique rules are enforced. sure, you could individually insist on only eating at healthy restaurants, and research each restaurant carefully before you go there. but it's just so much easier to have a defined plot of land where you have an assurance that anywhere you go within that plot of land will meet your standards. of course, you could lobby your local government to tighten health and safety regulations. but if you do that, you risk friction with people who have radically different preferences on tradeoffs, and you risk shutting poor people out of the economy. a network state offers a moderate approach. what is balaji's megapolitical case for network states? one of the curious features of the book that a reader will notice almost immediately is that it sometimes feels like two books in one: sometimes, it's a book about the concept of network states, and at other times it's an exposition of balaji's grand megapolitical theory. balaji's grand megapolitical theory is pretty out-there and fun in a bunch of ways. near the beginning of the book, he entices readers with tidbits like... ok fine, i'll just quote: germany sent vladimir lenin into russia, potentially as part of a strategy to destabilize their then-rival in war. antony sutton's books document how some wall street bankers apparently funded the russian revolution (and how other wall street bankers funded the nazis years later). leon trotsky spent time in new york prior to the revolution, and propagandistic reporting from americans like john reed aided lenin and trotsky in their revolution. indeed, reed was so useful to the soviets — and so misleading as to the nature of the revolution — that he was buried at the base of the kremlin wall. surprise: the russian revolution wasn't done wholly by russians, but had significant foreign involvement from germans and americans. the ochs-sulzberger family, which owns the new york times company, owned slaves but didn't report that fact in their 1619 coverage. new york times correspondent walter duranty won a pulitzer prize for helping the soviet union starve ukraine into submission, 90 years before the times decided to instead "stand with ukraine". you can find a bunch more juicy examples in the chapter titled, appropriately, "if the news is fake, imagine history". these examples seem haphazard, and indeed, to some extent they are so intentionally: the goal is first and foremost to shock the reader out of their existing world model so they can start downloading balaji's own. but pretty soon, balaji's examples do start to point to some particular themes: a deep dislike of the "woke" us left, exemplified by the new york times, a combination of strong discomfort with the chinese communist party's authoritarianism with an understanding of why the ccp often justifiably fears the united states, and an appreciation of the love of freedom of the us right (exemplified by bitcoin maximalists) combined with a dislike of their hostility toward cooperation and order. next, we get balaji's overview of the political realignments in recent history, and finally we get to his core model of politics in the present day: nyt, ccp, btc. team nyt basically runs the us, and its total lack of competence means that the us is collapsing. team btc (meaning, both actual bitcoin maximalists and us rightists in general) has some positive values, but their outright hostility to collective action and order means that they are incapable of building anything. team ccp can build, but they are building a dystopian surveillance state that much of the world would not want to live in. and all three teams are waaay too nationalist: they view things from the perspective of their own country, and ignore or exploit everyone else. even when the teams are internationalist in theory, their specific ways of interpreting their values make them unpalatable outside of a small part of the world. network states, in balaji's view, are a "de-centralized center" that could create a better alternative. they combine the love of freedom of team btc with the moral energy of team nyt and the organization of team ccp, and give us the best benefits of all three (plus a level of international appeal greater than any of the three) and avoid the worst parts. this is balajian megapolitics in a nutshell. it is not trying to justify network states using some abstract theory (eg. some dunbar's number or concentrated-incentive argument that the optimal size of a political body is actually in the low tens of thousands). rather, it is an argument that situates network states as a response to the particular political situation of the world at its current place and time. balaji's helical theory of history: yes, there are cycles, but there is also ongoing progress. right now, we're at the part of the cycle where we need to help the sclerotic old order die, but also seed a new and better one. do you have to agree with balaji's megapolitics to like network states? many aspects of balajian megapolitics will not be convincing to many readers. if you believe that "wokeness" is an important movement that protects the vulnerable, you may not appreciate the almost off-handed dismissal that it is basically just a mask for a professional elite's will-to-power. if you are worried about the plight of smaller countries such as ukraine who are threatened by aggressive neighbors and desperately need outside support, you will not be convinced by balaji's plea that "it may instead be best for countries to rearm, and take on their own defense". i do think that you can support network states while disagreeing with some of balaji's reasoning for them (and vice versa). but first, i should explain why i think balaji feels that his view of the problem and his view of the solution are connected. balaji has been passionate about roughly the same problem for a long time; you can see a similar narrative outline of defeating us institutional sclerosis through a technological and exit-driven approach in his speech on "the ultimate exit" from 2013. network states are the latest iteration of his proposed solution. there are a few reasons why talking about the problem is important: to show that network states are the only way to protect freedom and capitalism, one must show why the us cannot. if the us, or the "democratic liberal order", is just fine, then there is no need for alternatives; we should just double down on global coordination and rule of law. but if the us is in an irreversible decline, and its rivals are ascending, then things look quite different. network states can "maintain liberal values in an illiberal world"; hegemony thinking that assumes "the good guys are in charge" cannot. many of balaji's intended readers are not in the us, and a world of network states would inherently be globally distributed and that includes lots of people who are suspicious of america. balaji himself is indian, and has a large indian fan base. many people in india, and elsewhere, view the us not as a "guardian of the liberal world order", but as something much more hypocritical at best and sinister at worst. balaji wants to make it clear that you do not have to be pro-american to be a liberal (or at least a balaji-liberal). many parts of us left-leaning media are increasingly hostile to both cryptocurrency and the tech sector. balaji expects that the "authoritarian left" parts of "team nyt" will be hostile to network states, and he explains this by pointing out that the media are not angels and their attacks are often self-interested. but this is not the only way of looking at the broader picture. what if you do believe in the importance of role of social justice values, the new york times, or america? what if you value governance innovation, but have more moderate views on politics? then, there are two ways you could look at the issue: network states as a synergistic strategy, or at least as a backup. anything that happens in us politics in terms of improving equality, for example, only benefits the ~4% of the world's population that lives in the united states. the first amendment does not apply outside us borders. the governance of many wealthy countries is sclerotic, and we do need some way to try more governance innovation. network states could fill in the gaps. countries like the united states could host network states that attract people from all over the world. successful network states could even serve as a policy model for countries to adopt. alternatively, what if the republicans win and secure a decades-long majority in 2024, or the united states breaks down? you want there to be an alternative. exit to network states as a distraction, or even a threat. if everyone's first instinct when faced with a large problem within their country is to exit to an enclave elsewhere, there will be no one left to protect and maintain the countries themselves. global infrastructure that ultimately network states depend on will suffer. both perspectives are compatible with a lot of disagreement with balajian megapolitics. hence, to argue for or against balajian network states, we will ultimately have to talk about network states. my own view is friendly to network states, though with a lot of caveats and different ideas about how network states could work. what does cryptocurrency have to do with network states? there are two kinds of alignment here: there is the spiritual alignment, the idea that "bitcoin becomes the flag of technology", and there is the practical alignment, the specific ways in which network states could use blockchains and cryptographic tokens. in general, i agree with both of these arguments though i think balaji's book could do much more to spell them out more explicitly. the spiritual alignment cryptocurrency in 2022 is a key standard-bearer for internationalist liberal values that are difficult to find in any other social force that still stands strong today. blockchains and cryptocurrencies are inherently global. most ethereum developers are outside the us, living in far-flung places like europe, taiwan and australia. nfts have given unique opportunities to artists in africa and elsewhere in the global south. argentinians punch above their weight in projects like proof of humanity, kleros and nomic labs. blockchain communities continue to stand for openness, freedom, censorship resistance and credible neutrality, at a time where many geopolitical actors are increasingly only serving their own interests. this enhances their international appeal further: you don't have to love us hegemony to love blockchains and the values that they stand for. and this all makes blockchains an ideal spiritual companion for the network state vision that balaji wants to see. the practical alignment but spiritual alignment means little without practical use value for blockchains to go along with it. balaji gives plenty of blockchain use cases. one of balaji's favorite concepts is the idea of the blockchain as a "ledger of record": people can timestamp events on-chain, creating a global provable log of humanity's "microhistory". he continues with other examples: zero-knowledge technology like zcash, ironfish, and tornado cash allow on-chain attestation of exactly what people want to make public and nothing more. naming systems like the ethereum name service (ens) and solana name service (sns) attach identity to on-chain transactions. incorporation systems allow the on-chain representation of corporate abstractions above the level of a mere transaction, like financial statements or even full programmable company-equivalents like daos. cryptocredentials, non-fungible tokens (nfts), non-transferable fungibles (ntfs), and soulbounds allow the representation of non-financial data on chain, like diplomas or endorsements. but how does this all relate to network states? i could go into specific examples in the vein of crypto cities: issuing tokens, issuing citydao-style citizen nfts, combining blockchains with zero-knowledge cryptography to do secure privacy-preserving voting, and a lot more. blockchains are the lego of crypto-finance and crypto-governance: they are a very effective tool for implementing transparent in-protocol rules to govern common resources, assets and incentives. but we also need to go a level deeper. blockchains and network states have the shared property that they are both trying to "create a new root". a corporation is not a root: if there is a dispute inside a corporation, it ultimately gets resolved by a national court system. blockchains and network states, on the other hand, are trying to be new roots. this does not mean some absolute "na na no one can catch me" ideal of sovereignty that is perhaps only truly accessible to the ~5 countries that have highly self-sufficient national economies and/or nuclear weapons. individual blockchain participants are of course vulnerable to national regulation, and enclaves of network states even more so. but blockchains are the only infrastructure system that at least attempts to do ultimate dispute resolution at the non-state level (either through on-chain smart contract logic or through the freedom to fork). this makes them an ideal base infrastructure for network states. what aspects of balaji's vision do i like? given that a purist "private property rights only" libertarianism inevitably runs into large problems like its inability to fund public goods, any successful pro-freedom program in the 21st century has to be a hybrid containing at least one big compromise idea that solves at least 80% of the problems, so that independent individual initiative can take care of the rest. this could be some stringent measures against economic power and wealth concentration (maybe charge annual harberger taxes on everything), it could be an 85% georgist land tax, it could be a ubi, it could be mandating that sufficiently large companies become democratic internally, or one of any other proposals. not all of these work, but you need something that drastic to have any shot at all. generally, i am used to the big compromise idea being a leftist one: some form of equality and democracy. balaji, on the other hand, has big compromise ideas that feel more rightist: local communities with shared values, loyalty, religion, physical environments structured to encourage personal discipline ("keto kosher") and hard work. these values are implemented in a very libertarian and tech-forward way, organizing not around land, history, ethnicity and country, but around the cloud and personal choice, but they are rightist values nonetheless. this style of thinking is foreign to me, but i find it fascinating, and important. stereotypical "wealthy white liberals" ignore this at their peril: these more "traditional" values are actually quite popular even among some ethnic minorities in the united states, and even more so in places like africa and india, which is exactly where balaji is trying to build up his base. but what about this particular baizuo that's currently writing this review? do network states actually interest me? the "keto kosher" health-focused lifestyle immersion network state is certainly one that i would want to live in. sure, i could just spend time in cities with lots of healthy stuff that i can seek out intentionally, but a concentrated physical environment makes it so much easier. even the motivational aspect of being around other people who share a similar goal sounds very appealing. but the truly interesting stuff is the governance innovation: using network states to organize in ways that would actually not be possible under existing regulations. there are three ways that you can interpret the underlying goal here: creating new regulatory environments that let their residents have different priorities from the priorities preferred by the mainstream: for example, the "anyone can walk around naked" zone, or a zone that implements different tradeoffs between safety and convenience, or a zone that legalizes more psychoactive substances. creating new regulatory institutions that might be more efficient at serving the same priorities as the status quo. for example, instead of improving environmental friendliness by regulating specific behaviors, you could just have a pigovian tax. instead of requiring licenses and regulatory pre-approval for many actions, you could require mandatory liability insurance. you could use quadratic voting for governance and quadratic funding to fund local public goods. pushing against regulatory conservatism in general, by increasing the chance that there's some jurisdiction that will let you do any particular thing. institutionalized bioethics, for example, is a notoriously conservative enterprise, where 20 people dead in a medical experiment gone wrong is a tragedy, but 200000 people dead from life-saving medicines and vaccines not being approved quickly enough is a statistic. allowing people to opt into network states that accept higher levels of risk could be a successful strategy for pushing against this. in general, i see value in all three. a large-scale institutionalization of [1] could make the word simultaneously more free while making people comfortable with higher levels of restriction of certain things, because they know that if they want to do something disallowed there are other zones they could go to do it. more generally, i think there is an important idea hidden in [1]: while the "social technology" community has come up with many good ideas around better governance, and many good ideas around better public discussion, there is a missing emphasis on better social technology for sorting. we don't just want to take existing maps of social connections as given and find better ways to come to consensus within them. we also want to reform the webs of social connections themselves, and put people closer to other people that are more compatible with them to better allow different ways of life to maintain their own distinctiveness. [2] is exciting because it fixes a major problem in politics: unlike startups, where the early stage of the process looks somewhat like a mini version of the later stage, in politics the early stage is a public discourse game that often selects for very different things than what actually work in practice. if governance ideas are regularly implemented in network states, then we would move from an extrovert-privileging "talker liberalism" to a more balanced "doer liberalism" where ideas rise and fall based on how well they actually do on a small scale. we could even combine [1] and [2]: have a zone for people who want to automatically participate in a new governance experiment every year as a lifestyle. [3] is of course a more complicated moral question: whether you view paralysis and creep toward de-facto authoritarian global government as a bigger problem or someone inventing an evil technology that dooms us all as a bigger problem. i'm generally in the first camp; i am concerned about the prospect of both the west and china settling into a kind of low-growth conservatism, i love how imperfect coordination between nation states limits the enforceability of things like global copyright law, and i'm concerned about the possibility that, with future surveillance technology, the world as a whole will enter a highly self-enforcing but terrible political equilibrium that it cannot get out of. but there are specific areas (cough cough, unfriendly ai risk) where i am in the risk-averse camp ... but here we're already getting into the second part of my reaction. what aspects of balaji's vision do i take issue with? there are four aspects that i am worried about the most: the "founder" thing why do network states need a recognized founder to be so central? what if network states end up only serving the wealthy? "exit" alone is not sufficient to stabilize global politics. so if exit is everyone's first choice, what happens? what about global negative externalities more generally? the "founder" thing throughout the book, balaji is insistent on the importance of "founders" in a network state (or rather, a startup society: you found a startup society, and become a network state if you are successful enough to get diplomatic recognition). balaji explicitly describes startup society founders as being "moral entrepreneurs": these presentations are similar to startup pitch decks. but as the founder of a startup society, you aren't a technology entrepreneur telling investors why this new innovation is better, faster, and cheaper. you are a moral entrepreneur telling potential future citizens about a better way of life, about a single thing that the broader world has gotten wrong that your community is setting right. founders crystallize moral intuitions and learnings from history into a concrete philosophy, and people whose moral intuitions are compatible with that philosophy coalesce around the project. this is all very reasonable at an early stage though it is definitely not the only approach for how a startup society could emerge. but what happens at later stages? mark zuckerberg being the centralized founder of facebook the startup was perhaps necessary. but mark zuckerberg being in charge of a multibillion-dollar (in fact, multibillion-user) company is something quite different. or, for that matter, what about balaji's nemesis: the fifth-generation hereditary white ochs-sulzberger dynasty running the new york times? small things being centralized is great, extremely large things being centralized is terrifying. and given the reality of network effects, the freedom to exit again is not sufficient. in my view, the problem of how to settle into something other than founder control is important, and balaji spends too little effort on it. "recognized founder" is baked into the definition of what a balajian network state is, but a roadmap toward wider participation in governance is not. it should be. what about everyone who is not wealthy? over the last few years, we've seen many instances of governments around the world becoming explicitly more open to "tech talent". there are 42 countries offering digital nomad visas, there is a french tech visa, a similar program in singapore, golden visas for taiwan, a program for dubai, and many others. this is all great for skilled professionals and rich people. multimillionaires fleeing china's tech crackdowns and covid lockdowns (or, for that matter, moral disagreements with china's other policies) can often escape the world's systemic discrimination against chinese and other low-income-country citizens by spending a few hundred thousand dollars on buying another passport. but what about regular people? what about the rohingya minority facing extreme conditions in myanmar, most of whom do not have a way to enter the us or europe, much less buy another passport? here, we see a potential tragedy of the network state concept. on the one hand, i can really see how exit can be the most viable strategy for global human rights protection in the twenty first century. what do you do if another country is oppressing an ethnic minority? you could do nothing. you could sanction them (often ineffective and ruinous to the very people you're trying to help). you could try to invade (same criticism but even worse). exit is a more humane option. people suffering human rights atrocities could just pack up and leave for friendlier pastures, and coordinating to do it in a group would mean that they could leave without sacrificing the communities they depend on for friendship and economic livelihood. and if you're wrong and the government you're criticizing is actually not that oppressive, then people won't leave and all is fine, no starvation or bombs required. this is all beautiful and good. except... the whole thing breaks down because when the people try to exit, nobody is there to take them. what is the answer? honestly, i don't see one. one point in favor of network states is that they could be based in poor countries, and attract wealthy people from abroad who would then help the local economy. but this does nothing for people in poor countries who want to get out. good old-fashioned political action within existing states to liberalize immigration laws seems like the only option. nowhere to run in the wake of russia's invasion of ukraine on feb 24, noah smith wrote an important post on the moral clarity that the invasion should bring to our thought. a particularly striking section is titled "nowhere to run". quoting: but while exit works on a local level — if san francisco is too dysfunctional, you can probably move to austin or another tech town — it simply won't work at the level of nations. in fact, it never really did — rich crypto guys who moved to countries like singapore or territories like puerto rico still depended crucially on the infrastructure and institutions of highly functional states. but russia is making it even clearer that this strategy is doomed, because eventually there is nowhere to run. unlike in previous eras, the arm of the great powers is long enough to reach anywhere in the world. if the u.s. collapses, you can't just move to singapore, because in a few years you'll be bowing to your new chinese masters. if the u.s. collapses, you can't just move to estonia, because in a few years (months?) you'll be bowing to your new russian masters. and those masters will have extremely little incentive to allow you to remain a free individual with your personal fortune intact ... thus it is very very important to every libertarian that the u.s. not collapse. one possible counter-argument is: sure, if ukraine was full of people whose first instinct was exit, ukraine would have collapsed. but if russia was also more exit-oriented, everyone in russia would have pulled out of the country within a week of the invasion. putin would be left standing alone in the fields of the luhansk oblast facing zelensky a hundred meters away, and when putin shouts his demand for surrender, zelensky would reply: "you and what army"? (zelensky would of course win a fair one-on-one fight) but things could go a different way. the risk is that exitocracy becomes recognized as the primary way you do the "freedom" thing, and societies that value freedom will become exitocratic, but centralized states will censor and suppress these impulses, adopt a militaristic attitude of national unconditional loyalty, and run roughshod over everyone else. so what about those negative externalities? if we have a hundred much-less-regulated innovation labs everywhere around the world, this could lead to a world where harmful things are more difficult to prevent. this raises a question: does believing in balajism require believing in a world where negative externalities are not too big a deal? such a viewpoint would be the opposite of the vulnerable world hypothesis (vwh), which suggests that are technology progresses, it gets easier and easier for one or a few crazy people to kill millions, and global authoritarian surveillance might be required to prevent extreme suffering or even extinction. one way out might be to focus on self-defense technology. sure, in a network state world, we could not feasibly ban gain-of-function research, but we could use network states to help the world along a path to adopting really good hepa air filtering, far-uvc light, early detection infrastructure and a very rapid vaccine development and deployment pipeline that could defeat not only covid, but far worse viruses too. this 80,000 hours episode outlines the bull case for bioweapons being a solvable problem. but this is not a universal solution for all technological risks: at the very least, there is no self-defense against a super-intelligent unfriendly ai that kills us all. self-defense technology is good, and is probably an undervalued funding focus area. but it's not realistic to rely on that alone. transnational cooperation to, for example, ban slaughterbots, would be required. and so we do want a world where, even if network states have more sovereignty than intentional communities today, their sovereignty is not absolute. non-balajian network states reading the network state reminded me of a different book that i read ten years ago: david de ugarte's phyles: economic democracy in the twenty first century. phyles talks about similar ideas of transnational communities organized around values, but it has a much more left-leaning emphasis: it assumes that these communities will be democratic, inspired by a combination of 2000s-era online communities and nineteenth and twentieth-century ideas of cooperatives and workplace democracy. we can see the differences most clearly by looking at de ugarte's theory of formation. since i've already spent a lot of time quoting balaji, i'll give de ugarte a fair hearing with a longer quote: the very blogosphere is an ocean of identities and conversation in perpetual cross-breeding and change from among which the great social digestion periodically distils stable groups with their own contexts and specific knowledge. these conversational communities which crystallise, after a certain point in their development, play the main roles in what we call digital zionism: they start to precipitate into reality, to generate mutual knowledge among their members, which makes them more identitarially important to them than the traditional imaginaries of the imagined communities to which they are supposed to belong (nation, class, congregation, etc.) as if it were a real community (group of friends, family, guild, etc.) some of these conversational networks, identitarian and dense, start to generate their own economic metabolism, and with it a distinct demos – maybe several demoi – which takes the nurturing of the autonomy of the community itself as its own goal. these are what we call neo-venetianist networks. born in the blogosphere, they are heirs to the hacker work ethic, and move in the conceptual world, which tends to the economic democracy which we spoke about in the first part of this book. unlike traditional cooperativism, as they do not spring from real proximity-based communities, their local ties do not generate identity. in the indianos' foundation, for instance, there are residents in two countries and three autonomous regions, who started out with two companies founded hundreds of kilometres away from each other. we see some very balajian ideas: shared collective identities, but formed around values rather than geography, that start off as discussion communities in the cloud but then materialize into taking over large portions of economic life. de ugarte even uses the exact same metaphor ("digital zionism") that balaji does! but we also see a key difference: there is no single founder. rather than a startup society being formed by an act of a single individual combining together intuitions and strands of thought into a coherent formally documented philosophy, a phyle starts off as a conversational network in the blogosphere, and then directly turns into a group that does more and more over time all while keeping its democratic and horizontal nature. the whole process is much more organic, and not at all guided by a single person's intention. of course, the immediate challenge that i can see is the incentive issues inherent to such structures. one way to perhaps unfairly summarize both phyles and the network state is that the network state seeks to use 2010s-era blockchains as a model for how to reorganize human society, and phyles seeks to use 2000s-era open source software communities and blogs as a model for how to reorganize human society. open source has the failure mode of not enough incentives, cryptocurrency has the failure mode of excessive and overly concentrated incentives. but what this does suggest is that some kind of middle way should be possible. is there a middle way? my judgement so far is that network states are great, but they are far from being a viable big compromise idea that can actually plug all the holes needed to build the kind of world i and most of my readers would want to see in the 21st century. ultimately, i do think that we need to bring in more democracy and large-scale-coordination oriented big compromise ideas of some kind to make network states truly successful. here are some significant adjustments to balajism that i would endorse: founder to start is okay (though not the only way), but we really need a baked-in roadmap to exit-to-community many founders want to eventually retire or start something new (see: basically half of every crypto project), and we need to prevent network states from collapsing or sliding into mediocrity when that happens. part of this process is some kind of constitutional exit-to-community guarantee: as the network state enters higher tiers of maturity and scale, more input from community members is taken into account automatically. prospera attempted something like this. as scott alexander summarizes: once próspera has 100,000 residents (so realistically a long time from now, if the experiment is very successful), they can hold a referendum where 51% majority can change anything about the charter, including kicking hpi out entirely and becoming a direct democracy, or rejoining the rest of honduras, or anything but i would favor something even more participatory than the residents having an all-or-nothing nuclear option to kick the government out. another part of this process, and one that i've recognized in the process of ethereum's growth, is explicitly encouraging broader participation in the moral and philosophical development of the community. ethereum has its vitalik, but it also has its polynya: an internet anon who has recently entered the scene unsolicited and started providing high-quality thinking on rollups and scaling technology. how will your startup society recruit its first ten polynyas? network states should be run by something that's not coin-driven governance coin-driven governance is plutocratic and vulnerable to attacks; i have written about this many times, but it's worth repeating. ideas like optimism's soulbound and one-per-person citizen nfts are key here. balaji already acknowledges the need for non-fungibility (he supports coin lockups), but we should go further and more explicit in supporting governance that's not just shareholder-driven. this will also have the beneficial side effect that more democratic governance is more likely to be aligned with the outside world. network states commit to making themselves friendly through outside representation in governance one of the fascinating and under-discussed ideas from the rationalist and friendly-ai community is functional decision theory. this is a complicated concept, but the powerful core idea is that ais could coordinate better than humans, solving prisoner's dilemmas where humans often fail, by making verifiable public commitments about their source code. an ai could rewrite itself to have a module that prevents it from cheating other ais that have a similar module. such ais would all cooperate with each other in prisoner's dilemmas. as i pointed out years ago, daos could potentially do the same thing. they could have governance mechanisms that are explicitly more charitable toward other daos that have a similar mechanism. network states would be run by daos, and this would apply to network states too. they could even commit to governance mechanisms that promise to take wider public interests into account (eg. 20% of the votes could go to a randomly selected set of residents of the host city or country), without the burden of having to follow specific complicated regulations of how they should take those interests into account. a world where network states do such a thing, and where countries adopt policies that are explicitly more friendly to network states that do it, could be a better one. conclusion i want to see startup societies along these kinds of visions exist. i want to see immersive lifestyle experiments around healthy living. i want to see crazy governance experiments where public goods are funded by quadratic funding, and all zoning laws are replaced by a system where every building's property tax floats between zero and five percent per year based on what percentage of nearby residents express approval or disapproval in a real-time blockchain and zkp-based voting system. and i want to see more technological experiments that accept higher levels of risk, if the people taking those risks consent to it. and i think blockchain-based tokens, identity and reputation systems and daos could be a great fit. at the same time, i worry that the network state vision in its current form risks only satisfying these needs for those wealthy enough to move and desirable enough to attract, and many people lower down the socioeconomic ladder will be left in the dust. what can be said in network states' favor is their internationalism: we even have the africa-focused afropolitan. inequalities between countries are responsible for two thirds of global inequality and inequalities within countries are only one third. but that still leaves a lot of people in all countries that this vision doesn't do much for. so we need something else too for the global poor, for ukrainians that want to keep their country and not just squeeze into poland for a decade until poland gets invaded too, and everyone else that's not in a position to move to a network state tomorrow or get accepted by one. network states, with some modifications that push for more democratic governance and positive relationships with the communities that surround them, plus some other way to help everyone else? that is a vision that i can get behind. dark mode toggle blockchain voting is overrated among uninformed people but underrated among informed people 2021 may 25 see all posts special thanks to karl floersch, albert ni, mr silly and others for feedback and discussion voting is a procedure that has a very important need for process integrity. the result of the vote must be correct, and this must be guaranteed by a transparent process so that everyone can be convinced that the result is correct. it should not be possible to successfully interfere with anyone's attempt to vote or prevent their vote from being counted. blockchains are a technology which is all about providing guarantees about process integrity. if a process is run on a blockchain, the process is guaranteed to run according to some pre-agreed code and provide the correct output. no one can prevent the execution, no one can tamper with the execution, and no one can censor and block any users' inputs from being processed. so at first glance, it seems that blockchains provide exactly what voting needs. and i'm far from the only person to have had that thought; plenty of major prospective users are interested. but as it turns out, some people have a very different opinion.... despite the seeming perfect match between the needs of voting and the technological benefits that blockchains provide, we regularly see scary articles arguing against the combination of the two. and it's not just a single article: here's an anti-blockchain-voting piece from scientific american, here's another from cnet, and here's another from arstechnica. and it's not just random tech journalists: bruce schneier is against blockchain voting, and researchers at mit wrote a whole paper arguing that it's a bad idea. so what's going on? outline there are two key lines of criticism that are most commonly levied by critics of blockchain voting protocols: blockchains are the wrong software tool to run an election. the trust properties they provide are not a good match for the properties that voting needs, and other kinds of software tools with different information flow and trust properties would work better. software in general cannot be trusted to run elections, no matter what software it is. the risk of undetectable software and hardware bugs is too high, no matter how the platform is organized. this article will discuss both of these claims in turn ("refute" is too strong a word, but i definitely disagree more than i agree with both claims). first, i will discuss the security issues with existing attempts to use blockchains for voting, and how the correct solution is not to abandon blockchains, but to combine them with other cryptographic technologies. second, i will address the concern about whether or not software (and hardware) can be trusted. the answer: computer security is actually getting quite a bit better, and we can work hard to continue that trend. over the long term, insisting on paper permanently would be a huge handicap to our ability to make voting better. one vote per n years is a 250-year-old form of democracy, and we can have much better democracy if voting were much more convenient and simpler, so that we could do it much more often. needless to say, this entire post is predicated on good blockchain scaling technology (eg. sharding) being available. of course, if blockchains cannot scale, none of this can happen. but so far, development of this technology is proceeding quickly, and there's no reason to believe that it can't happen. bad blockchain voting protocols blockchain voting protocols get hacked all the time. two years ago, a blockchain voting tech company called voatz was all the rage, and many people were very excited about it. but last year, some mit researchers discovered a string of critical security vulnerabilities in their platform. meanwhile, in moscow, a blockchain voting system that was going to be used for an upcoming election was hacked, fortunately a month before the election took place. the hacks were pretty serious. here is a table of the attack capabilities that researchers analyzing voatz managed to uncover: this by itself is not an argument against ever using blockchain voting. but it is an argument that blockchain voting software should be designed more carefully, and scaled up slowly and incrementally over time. privacy and coercion resistance but even the blockchain voting protocols that are not technically broken often suck. to understand why, we need to delve deeper into what specific security properties blockchains provide, and what specific security properties voting needs when we do, we'll see that there is a mismatch. blockchains provide two key properties: correct execution and censorship resistance. correct execution just means that the blockchain accepts inputs ("transactions") from users, correctly processes them according to some pre-defined rules, and returns the correct output (or adjusts the blockchain's "state" in the correct way). censorship resistance is also simple to understand: any user that wants to send a transaction, and is willing to pay a high enough fee, can send the transaction and expect to see it quickly included on-chain. both of these properties are very important for voting: you want the output of the vote to actually be the result of counting up the number of votes for each candidate and selecting the candidate with the most votes, and you definitely want anyone who is eligible to vote to be able to vote, even if some powerful actor is trying to block them. but voting also requires some crucial properties that blockchains do not provide: privacy: you should not be able to tell which candidate someone specific voted for, or even if they voted at all coercion resistance: you should not be able to prove to someone else how you voted, even if you want to the need for the first requirement is obvious: you want people to vote based on their personal feelings, and not how people around them or their employer or the police or random thugs on the street will feel about their choice. the second requirement is needed to prevent vote selling: if you can prove how you voted, selling your vote becomes very easy. provability of votes would also enable forms of coercion where the coercer demands to see some kind of proof of voting for their preferred candidate. most people, even those aware of the first requirement, do not think about the second requirement. but the second requirement is also necessary, and it's quite technically nontrivial to provide it. needless to say, the average "blockchain voting system" that you see in the wild does not even try to provide the second property, and usually fails at providing the first. secure electronic voting without blockchains the concept of cryptographically secured execution of social mechanisms was not invented by blockchain geeks, and indeed existed far before us. outside the blockchain space, there is a 20-year-old tradition of cryptographers working on the secure electronic voting problem, and the good news is that there have been solutions. an important paper that is cited by much of the literature of the last two decades is juels, catalano and jakobsson's 2002 paper titled "coercion-resistant electronic elections": since then, there have been many iterations on the concept; civitas is one prominent example, though there are also many others. these protocols all use a similar set of core techniques. there is an agreed-upon set of "talliers" and there is a trust assumption that the majority of the talliers is honest. the talliers each have "shares" of a private key secret-shared among themselves, and the corresponding public key is published. voters publish votes encrypted to the talliers' public key, and talliers use a secure multi-party computation (mpc) protocol to decrypt and verify the votes and compute the tally. the tallying computation is done "inside the mpc": the talliers never learn their private key, and they compute the final result without learning anything about any individual vote beyond what can be learned from looking at the final result itself. encrypting votes provides privacy, and some additional infrastructure such as mix-nets is added on top to make the privacy stronger. to provide coercion resistance, one of two techniques is used. one option is that during the registration phase (the phase in which the talliers learn each registered voter's public key), the voter generates or receives a secret key. the corresponding public key is secret shared among the talliers, and the talliers' mpc only counts a vote if it is signed with the secret key. a voter has no way to prove to a third party what their secret key is, so if they are bribed or coerced they can simply show and cast a vote signed with the wrong secret key. alternatively, a voter could have the ability to send a message to change their secret key. a voter has no way of proving to a third party that they did not send such a message, leading to the same result. the second option is a technique where voters can make multiple votes where the second overrides the first. if a voter is bribed or coerced, they can make a vote for the briber/coercer's preferred candidate, but later send another vote to override the first. giving voters the ability to make a later vote that can override an earlier vote is the key coercion-resistance mechanism of this protocol from 2015. now, we get to a key important nuance in all of these protocols. they all rely on an outside primitive to complete their security guarantees: the bulletin board (this is the "bb" in the figure above). the bulletin board is a place where any voter can send a message, with a guarantee that (i) anyone can read the bulletin board, and (ii) anyone can send a message to the bulletin board that gets accepted. most of the coercion-resistant voting papers that you can find will casually reference the existence of a bulletin board (eg. "as is common for electronic voting schemes, we assume a publicly accessible append-only bulletin board"), but far fewer papers talk about how this bulletin board can actually be implemented. and here, you can hopefully see where i am going with this: the most secure way to implement a bulletin board is to just use an existing blockchain! secure electronic voting with blockchains of course, there have been plenty of pre-blockchain attempts at making a bulletin board. this paper from 2008 is such an attempt; its trust model is a standard requirement that "k of n servers must be honest" (k = n/2 is common). this literature review from 2021 covers some pre-blockchain attempts at bulletin boards as well as exploring the use of blockchains for the job; the pre-blockchain solutions reviewed similarly rely on a k-of-n trust model. a blockchain is also a k-of-n trust model; it requires at least half of miners or proof of stake validators to be following the protocol, and if that assumption fails that often results in a "51% attack". so why is a blockchain better than a special purpose bulletin board? the answer is: setting up a k-of-n system that's actually trusted is hard, and blockchains are the only system that has already solved it, and at scale. suppose that some government announced that it was making a voting system, and provided a list of 15 local organizations and universities that would be running a special-purpose bulletin board. how would you, as an outside observer, know that the government didn't just choose those 15 organizations from a list of 1000 based on their willingness to secretly collude with an intelligence agency? public blockchains, on the other hand, have permissionless economic consensus mechanisms (proof of work or proof of stake) that anyone can participate in, and they have an existing diverse and highly incentivized infrastructure of block explorers, exchanges and other watching nodes to constantly verify in real time that nothing bad is going on. these more sophisticated voting systems are not just using blockchains; they rely on cryptography such as zero knowledge proofs to guarantee correctness, and on multi-party computation to guarantee coercion resistance. hence, they avoid the weaknesses of more naive systems that simply just "put votes directly on the blockchain" and ignore the resulting privacy and coercion resistance issues. however, the blockchain bulletin board is nevertheless a key part of the security model of the whole design: if the committee is broken but the blockchain is not, coercion resistance is lost but all the other guarantees around the voting process still remain. maci: coercion-resistant blockchain voting in ethereum the ethereum ecosystem is currently experimenting with a system called maci that combines together a blockchain, zk-snarks and a single central actor that guarantees coercion resistance (but has no power to compromise any properties other than coercion resistance). maci is not very technically difficult. users participate by signing a message with their private key, encrypting the signed message to a public key published by a central server, and publishing the encrypted signed message to the blockchain. the server downloads the messages from the blockchain, decrypts them, processes them, and outputs the result along with a zk-snark to ensure that they did the computation correctly. users cannot prove how they participated, because they have the ability to send a "key change" message to trick anyone trying to audit them: they can first send a key change message to change their key from a to b, and then send a "fake message" signed with a. the server would reject the message, but no one else would have any way of knowing that the key change message had ever been sent. there is a trust requirement on the server, though only for privacy and coercion resistance; the server cannot publish an incorrect result either by computing incorrectly or by censoring messages. in the long term, multi-party computation can be used to decentralize the server somewhat, strengthening the privacy and coercion resistance guarantees. there is a working demo of this scheme at clr.fund being used for quadratic funding. the use of the ethereum blockchain to ensure censorship resistance of votes ensures a much higher degree of censorship resistance than would be possible if a committee was relied on for this instead. recap the voting process has four important security requirements that must be met for a vote to be secure: correctness, censorship resistance, privacy and coercion resistance. blockchains are good at the first two. they are bad at the last two. encryption of votes put on a blockchain can add privacy. zero knowledge proofs can bring back correctness despite observers being unable to add up votes directly because they are encrypted. multi-party computation decrypting and checking votes can provide coercion resistance, if combined with a mechanic where users can interact with the system multiple times; either the first interaction invalidates the second, or vice versa using a blockchain ensures that you have very high-security censorship resistance, and you keep this censorship resistance even if the committee colludes and breaks coercion resistance. introducing a blockchain can significantly increase the level of security of the system. but can technology be trusted? but now we get back to the second, deeper, critique of electronic voting of any kind, blockchain or not: that technology itself is too insecure to be trusted. the recent mit paper criticizing blockchain voting includes this helpful table, depicting any form of paperless voting as being fundamentally too difficult to secure: the key property that the authors focus on is software-independence, which they define as "the property that an undetected change or error in a system's software cannot cause an undetectable change in the election outcome". basically, a bug in the code should not be able to accidentally make prezzy mcpresidentface the new president of the country (or, more realistically, a deliberately inserted bug should not be able to increase some candidate's share from 42% to 52%). but there are other ways to deal with bugs. for example, any blockchain-based voting system that uses publicly verifiable zero-knowledge proofs can be independently verified. someone can write their own implementation of the proof verifier and verify the zk-snark themselves. they could even write their own software to vote. of course, the technical complexity of actually doing this is beyond 99.99% of any realistic voter base, but if thousands of independent experts have the ability to do this and verify that it works, that is more than good enough in practice. to the mit authors, however, that is not enough: thus, any system that is electronic only, even if end-to-end verifiable, seems unsuitable for political elections in the foreseeable future. the u.s. vote foundation has noted the promise of e2e-v methods for improving online voting security, but has issued a detailed report recommending avoiding their use for online voting unless and until the technology is far more mature and fully tested in pollsite voting [38]. others have proposed extensions of these ideas. for example, the proposal of juels et al. [55] emphasizes the use of cryptography to provide a number of forms of "coercion resistance." the civitas proposal of clarkson et al. [24] implements additional mechanisms for coercion resistance, which iovino et al. [53] further incorporate and elaborate into their selene system. from our perspective, these proposals are innovative but unrealistic: they are quite complex, and most seriously, their security relies upon voters' devices being uncompromised and functioning as intended, an unrealistic assumption. the problem that the authors focus on is not the voting system's hardware being secure; risks on that side actually can be mitigated with zero knowledge proofs. rather, the authors focus on a different security problem: can users' devices even in principle be made secure? given the long history of all kinds of exploits and hacks of consumer devices, one would be very justified in thinking the answer is "no". quoting my own article on bitcoin wallet security from 2013: last night around 9pm pdt, i clicked a link to go to coinchat[.]freetzi[.]com – and i was prompted to run java. i did (thinking this was a legitimate chatoom), and nothing happened. i closed the window and thought nothing of it. i opened my bitcoin-qt wallet approx 14 minutes later, and saw a transaction that i did not approve go to wallet 1es3qvvkn1qa2p6me7jlcvmzpqxvxwpntc for almost my entire wallet... and: in june 2011, the bitcointalk member "allinvain" lost 25,000 btc (worth $500,000 at the time) after an unknown intruder somehow gained direct access to his computer. the attacker was able to access allinvain's wallet.dat file, and quickly empty out the wallet – either by sending a transaction from allinvain's computer itself, or by simply uploading the wallet.dat file and emptying it on his own machine. but these disasters obscure a greater truth: over the past twenty years, computer security has actually been slowly and steadily improving. attacks are much harder to find, often requiring the attacker to find bugs in multiple sub-systems instead of finding a single hole in a large complex piece of code. high-profile incidents are larger than ever, but this is not a sign that anything is getting less secure; rather, it's simply a sign that we are becoming much more dependent on the internet. trusted hardware is a very important recent source of improvements. some of the new "blockchain phones" (eg. this one from htc) go quite far with this technology and put a minimalistic security-focused operating system on the trusted hardware chip, allowing high-security-demanding applications (eg. cryptocurrency wallets) to stay separate from the other applications. samsung has started making phones using similar technology. and even devices that are never advertised as "blockchain devices" (eg. iphones) frequently have trusted hardware of some kind. cryptocurrency hardware wallets are effectively the same thing, except the trusted hardware module is physically located outside the computer instead of inside it. trusted hardware (deservedly!) often gets a bad rap in security circles and especially the blockchain community, because it just keeps getting broken again and again. and indeed, you definitely don't want to use it to replace your security protection. but as an augmentation, it's a huge improvement. finally, single applications, like cryptocurrency wallets and voting systems, are much simpler and have less room for error than an entire consumer operating system even if you have to incorporate support for quadratic voting, sortition, quadratic sortition and whatever horrors the next generation's glen weyl invents in 2040. the benefit of tools like trusted hardware is their ability to isolate the simple thing from the complex and possibly broken thing, and these tools are having some success. so the risks might decrease over time. but what are the benefits? these improvements in security technology point to a future where consumer hardware might be more trusted in the future than it is today. investments made in this area in the last few years are likely to keep paying off over the next decade, and we could expect further significant improvements. but what are the benefits of making voting electronic (blockchain based or otherwise) that justify exploring this whole space? my answer is simple: voting would become much more efficient, allowing us to do it much more often. currently, formal democratic input into organizations (governmental or corporate) tends to be limited to a single vote once every 1-6 years. this effectively means that each voter is only putting less than one bit of input into the system each year. perhaps in large part as a result of this, decentralized decision-making in our society is heavily bifurcated into two extremes: pure democracy and pure markets. democracy is either very inefficient (corporate and government votes) or very insecure (social media likes/retweets). markets are far more technologically efficient and are much more secure than social media, but their fundamental economic logic makes them a poor fit for many kinds of decision problems, particularly having to do with public goods. yes, i know it's yet another triangle, and i really really apologize for having to use it. but please bear with me just this once.... (ok fine, i'm sure i'll make even more triangles in the future; just suck it up and deal with it) there is a lot that we could do if we could build more systems that are somewhere in between democracy and markets, benefiting from the egalitarianism of the former, the technical efficiency of the latter and economic properties all along the spectrum in between the two extremes. quadratic funding is an excellent example of this. liquid democracy is another excellent example. even if we don't introduce fancy new delegation mechanisms or quadratic math, there's a lot that we could do by doing voting much more and at smaller scales more adapted to the information available to each individual voter. but the challenge with all of these ideas is that in order to have a scheme that durably maintains any level of democraticness at all, you need some form of sybil resistance and vote-buying mitigation: exactly the problems that these fancy zk-snark + mpc + blockchain voting schemes are trying to solve. the crypto space can help one of the underrated benefits of the crypto space is that it's an excellent "virtual special economic zone" for testing out economic and cryptographic ideas in a highly adversarial environment. whatever you build and release, once the economic power that it controls gets above a certain size, a whole host of diverse, sometimes altruistic, sometimes profit-motivated, and sometimes malicious actors, many of whom are completely anonymous, will descend upon the system and try to twist that economic power toward their own various objectives. the incentives for attackers are high: if an attacker steals $100 from your cryptoeconomic gadget, they can often get the full $100 in reward, and they can often get away with it. but the incentives for defenders are also high: if you develop a tool that helps users not lose their funds, you could (at least sometimes) turn that into a tool and earn millions. crypto is the ultimate training zone: if you can build something that can survive in this environment at scale, it can probably also survive in the bigger world as well. this applies to quadratic funding, it applies to multisig and social recovery wallets, and it can apply to voting systems too. the blockchain space has already helped to motivate the rise of important security technologies: hardware wallets efficient general-purpose zero knowledge proofs formal verification tools "blockchain phones" with trusted hardware chips anti-sybil schemes like proof of humanity in all of these cases, some version of the technology existed before blockchains came onto the scene. but it's hard to deny that blockchains have had a significant impact in pushing these efforts forward, and the large role of incentives inherent to the space plays a key role in raising the stakes enough for the development of the tech to actually happen. conclusion in the short term, any form of blockchain voting should certainly remain confined to small experiments, whether in small trials for more mainstream applications or for the blockchain space itself. security is at present definitely not good enough to rely on computers for everything. but it's improving, and if i am wrong and security fails to improve then not only blockchain voting, but also cryptocurrency as a whole, will have a hard time being successful. hence, there is a large incentive for the technology to continue to improve. we should all continue watching the technology and the efforts being made everywhere to try and increase security, and slowly become more comfortable using technology in very important social processes. technology is already key in our financial markets, and a crypto-ization of a large part of the economy (or even just replacing gold) will put an even greater portion of the economy into the hands of our cryptographic algorithms and the hardware that is running them. we should watch and support this process carefully, and over time take advantage of its benefits to bring our governance technologies into the 21st century. dark mode toggle some ways to use zk-snarks for privacy 2022 jun 15 see all posts special thanks to barry whitehat and gubsheep for feedback and review. zk-snarks are a powerful cryptographic tool, and an increasingly important part of the applications that people are building both in the blockchain space and beyond. but they are complicated, both in terms of how they work, and in terms of how you can use them. my previous post explaining zk-snarks focused on the first question, attempting to explain the math behind zk-snarks in a way that's reasonably understandable but still theoretically complete. this post will focus on the second question: how do zk-snarks fit into existing applications, what are some examples of what they can do, what can't they do, and what are some general guidelines for figuring out whether or not zk-snarking some particular application is possible? in particular, this post focuses on applications of zk-snarks for preserving privacy. what does a zk-snark do? suppose that you have a public input \(x\), a private input \(w\), and a (public) function \(f(x, w) \rightarrow \{true, false\}\) that performs some kind of verification on the inputs. with a zk-snark, you can prove that you know an \(w\) such that \(f(x, w) = true\) for some given \(f\) and \(x\), without revealing what \(w\) is. additionally, the verifier can verify the proof much faster it would take for them to compute \(f(x, w)\) themselves, even if they know \(w\). this gives the zk-snark its two properties: privacy and scalability. as mentioned above, in this post our examples will focus on privacy. proof of membership suppose that you have an ethereum wallet, and you want to prove that this wallet has a proof-of-humanity registration, without revealing which registered human you are. we can mathematically describe the function as follows: the private input (\(w\)): your address \(a\), and the private key \(k\) to your address the public input (\(x\)): the set of all addresses with verified proof-of-humanity profiles \(\{h_1 ... h_n\}\) the verification function \(f(x, w)\): interpret \(w\) as the pair \((a, k)\), and \(x\) as the list of valid profiles \(\{h_1 ... h_n\}\) verify that \(a\) is one of the addresses in \(\{h_1 ... h_n\}\) verify that \(privtoaddr(k) = a\) return \(true\) if both verifications pass, \(false\) if either verification fails the prover generates their address \(a\) and the associated key \(k\), and provides \(w = (a, k)\) as the private input to \(f\). they take the public input, the current set of verified proof-of-humanity profiles \(\{h_1 ... h_n\}\), from the chain. they run the zk-snark proving algorithm, which (assuming the inputs are correct) generates the proof. the prover sends the proof to the verifier and they provide the block height at which they obtained the list of verified profiles. the verifier also reads the chain, gets the list \(\{h_1 ... h_n\}\) at the height that the prover specified, and checks the proof. if the check passes, the verifier is convinced that the prover has some verified proof-of-humanity profile. before we move on to more complicated examples, i highly recommend you go over the above example until you understand every bit of what is going on. making the proof-of-membership more efficient one weakness in the above proof system is that the verifier needs to know the whole set of profiles \(\{h_1 ... h_n\}\), and they need to spend \(o(n)\) time "inputting" this set into the zk-snark mechanism. we can solve this by instead passing in as a public input an on-chain merkle root containing all profiles (this could just be the state root). we add another private input, a merkle proof \(m\) proving that the prover's account \(a\) is in the relevant part of the tree. advanced readers: a very new and more efficient alternative to merkle proofs for zk-proving membership is caulk. in the future, some of these use cases may migrate to caulk-like schemes. zk-snarks for coins projects like zcash and tornado.cash allow you to have privacy-preserving currency. now, you might think that you can take the "zk proof-of-humanity" above, but instead of proving access of a proof-of-humanity profile, use it to prove access to a coin. but we have a problem: we have to simultaneously solve privacy and the double spending problem. that is, it should not be possible to spend the coin twice. here's how we solve this. anyone who has a coin has a private secret \(s\). they locally compute the "leaf" \(l = hash(s, 1)\), which gets published on-chain and becomes part of the state, and \(n = hash(s, 2)\), which we call the nullifier. the state gets stored in a merkle tree. to spend a coin, the sender must make a zk-snark where: the public input contains a nullifier \(n\), the current or recent merkle root \(r\), and a new leaf \(l'\) (the intent is that recipient has a secret \(s'\), and passes to the sender \(l' = hash(s', 1)\)) the private input contains a secret \(s\), a leaf \(l\) and a merkle branch \(m\) the verification function checks that: \(m\) is a valid merkle branch proving that \(l\) is a leaf in a tree with root \(r\), where \(r\) is the current merkle root of the state \(hash(s, 1) = l\) \(hash(s, 2) = n\) the transaction contains the nullifier \(n\) and the new leaf \(l'\). we don't actually prove anything about \(l'\), but we "mix it in" to the proof to prevent \(l'\) from being modified by third parties when the transaction is in-flight. to verify the transaction, the chain checks the zk-snark, and additionally checks that \(n\) has not been used in a previous spending transaction. if the transaction succeeds, \(n\) is added to the spent nullifier set, so that it cannot be spent again. \(l'\) is added to the merkle tree. what is going on here? we are using a zk-snark to relate two values, \(l\) (which goes on-chain when a coin is created) and \(n\) (which goes on-chain when a coin is spent), without revealing which \(l\) is connected to which \(n\). the connection between \(l\) and \(n\) can only be discovered if you know the secret \(s\) that generates both. each coin that gets created can only be spent once (because for each \(l\) there is only one valid corresponding \(n\)), but which coin is being spent at a particular time is kept hidden. this is also an important primitive to understand. many of the mechanisms we describe below will be based on a very similar "privately spend only once" gadget, though for different purposes. coins with arbitrary balances the above can easily be extended to coins of arbitrary balances. we keep the concept of "coins", except each coin has a (private) balance attached. one simple way to do this is have the chain store for each coin not just the leaf \(l\) but also an encrypted balance. each transaction would consume two coins and create two new coins, and it would add two (leaf, encrypted balance) pairs to the state. the zk-snark would also check that the sum of the balances coming in equals the sum of the balances going out, and that the two output balances are both non-negative. zk anti-denial-of-service an interesting anti-denial-of-service gadget. suppose that you have some on-chain identity that is non-trivial to create: it could be a proof-of-humanity profile, it could be a validator with 32 eth, or it could just be an account that has a nonzero eth balance. we could create a more dos resistant peer-to-peer network by only accepting a message if it comes with a proof that the message's sender has such a profile. every profile would be allowed to send up to 1000 messages per hour, and a sender's profile would be removed from the list if the sender cheats. but how do we make this privacy-preserving? first, the setup. let \(k\) be the private key of a user; \(a = privtoaddr(k)\) is the corresponding address. the list of valid addresses is public (eg. it's a registry on-chain). so far this is similar to the proof-of-humanity example: you have to prove that you have the private key to one address without revealing which one. but here, we don't just want a proof that you're in the list. we want a protocol that lets you prove you're in the list but prevents you from making too many proofs. and so we need to do some more work. we'll divide up time into epochs; each epoch lasts 3.6 seconds (so, 1000 epochs per hour). our goal will be to allow each user to send only one message per epoch; if the user sends two messages in the same epoch, they will get caught. to allow users to send occasional bursts of messages, they are allowed to use epochs in the recent past, so if some user has 500 unused epochs they can use those epochs to send 500 messages all at once. the protocol we'll start with a simple version: we use nullifiers. a user generates a nullifier with \(n = hash(k, e)\), where \(k\) is their key and \(e\) is the epoch number, and publishes it along with the message \(m\). the zk-snark once again mixes in \(hash(m)\) without verifying anything about \(m\), so that the proof is bound to a single message. if a user makes two proofs bound to two different messages with the same nullifier, they can get caught. now, we'll move on to the more complex version. instead of just making it easy to prove if someone used the same epoch twice, this next protocol will actually reveal their private key in that case. our core technique will rely on the "two points make a line" trick: if you reveal one point on a line, you've revealed little, but if you reveal two points on a line, you've revealed the whole line. for each epoch \(e\), we take the line \(l_e(x) = hash(k, e) * x + k\). the slope of the line is \(hash(k, e)\), and the y-intercept is \(k\); neither is known to the public. to make a certificate for a message \(m\), the sender provides \(y = l_e(hash(m)) =\) \(hash(k, e) * hash(m) + k\), along with a zk-snark proving that \(y\) was computed correctly. to recap, the zk-snark here is as follows: public input: \(\{a_1 ... a_n\}\), the list of valid accounts \(m\), the message that the certificate is verifying \(e\), the epoch number used for the certificate \(y\), the line function evaluation private input: \(k\), your private key verification function: check that \(privtoaddr(k)\) is in \(\{a_1 ... a_n\}\) check that \(y = hash(k, e) * hash(m) + k\) but what if someone uses a single epoch twice? that means they published two values \(m_1\) and \(m_2\) and the corresponding certificate values \(y_1 = hash(k, e) * hash(m_1) + k\) and \(y_2 = hash(k, e) * hash(m_2) + k\). we can use the two points to recover the line, and hence the y-intercept (which is the private key): \(k = y_1 hash(m_1) * \frac{y_2 y_1}{hash(m_2) hash(m_1)}\) so if someone reuses an epoch, they leak out their private key for everyone to see. depending on the circumstance, this could imply stolen funds, a slashed validator, or simply the private key getting broadcasted and included into a smart contract, at which point the corresponding address would get removed from the set. what have we accomplished here? a viable off-chain, anonymous anti-denial-of-service system useful for systems like blockchain peer-to-peer networks, chat applications, etc, without requiring any proof of work. the rln (rate limiting nullifier) project is currently building essentially this idea, though with minor modifications (namely, they do both the nullifier and the two-points-on-a-line technique, using the nullifier to make it easier to catch double-use of an epoch). zk negative reputation suppose that we want to build 0chan, an internet forum which provides full anonymity like 4chan (so you don't even have persistent names), but has a reputation system to encourage more quality content. this could be a system where some moderation dao can flag posts as violating the rules of the system and institutes a three-strikes-and-you're-out mechanism, it could be users being able to upvote and downvote posts; there are lots of configurations. the reputation system could support positive or negative reputation; however, supporting negative reputation requires extra infrastructure to require the user to take into account all reputation messages in their proof, even the negative ones. it's this harder use case, which is similar to what is being implemented with unirep social, that we'll focus on. chaining posts: the basics anyone can make a post by publishing a message on-chain that contains the post, and a zk-snark proving that either (i) you own some scarce external identity, eg. proof-of-humanity, that entitles you to create an account, or (ii) that you made some specific previous post. specifically, the zk-snark is as follows: public inputs: the nullifier \(n\) a recent blockchain state root \(r\) the post contents ("mixed in" to the proof to bind it to the post, but we don't do any computation on it) private inputs: your private key \(k\) either an external identity (with address \(a\)), or the nullifier \(n_{prev}\) used by the previous post a merkle proof \(m\) proving inclusion of \(a\) or \(n_{prev}\) on-chain the number \(i\) of posts that you have previously made with this account verification function: check that \(m\) is a valid merkle branch proving that (either \(a\) or \(n_{prev}\), whichever is provided) is a leaf in a tree with root \(r\) check that \(n = enc(i, k)\), where \(enc\) is an encryption function (eg. aes) if \(i = 0\), check that \(a = privtoaddr(k)\), otherwise check that \(n_{prev} = enc(i-1, k)\) in addition to verifying the proof, the chain also checks that (i) \(r\) actually is a recent state root, and (ii) the nullifier \(n\) has not yet been used. so far, this is like the privacy-preserving coin introduced earlier, but we add a procedure for "minting" a new account, and we remove the ability to "send" your account to a different key instead, all nullifiers are generated using your original key. we use \(enc\) instead of \(hash\) here to make the nullifiers reversible: if you have \(k\), you can decrypt any specific nullifier you see on-chain and if the result is a valid index and not random junk (eg. we could just check \(dec(n) < 2^{64}\)), then you know that nullifier was generated using \(k\). adding reputation reputation in this scheme is on-chain and in the clear: some smart contract has a method addreputation, which takes as input (i) the nullifier published along with the post, and (ii) the number of reputation units to add and subtract. we extend the on-chain data stored per post: instead of just storing the nullifier \(n\), we store \(\{n, \bar{h}, \bar{u}\}\), where: \(\bar{h} = hash(h, r)\) where \(h\) is the block height of the state root that was referenced in the proof \(\bar{u} = hash(u, r)\) where \(u\) is the account's reputation score (0 for a fresh account) \(r\) here is simply a random value, added to prevent \(h\) and \(u\) from being uncovered by brute-force search (in cryptography jargon, adding \(r\) makes the hash a hiding commitment). suppose that a post uses a root \(r\) and stores \(\{n, \bar{h}, \bar{u}\}\). in the proof, it links to a previous post, with stored data \(\{n_{prev}, \bar{h}_{prev}, \bar{u}_{prev}\}\). the post's proof is also required to walk over all the reputation entries that have been published between \(h_{prev}\) and \(h\). for each nullifier \(n\), the verification function would decrypt \(n\) using the user's key \(k\), and if the decryption outputs a valid index it would apply the reputation update. if the sum of all reputation updates is \(\delta\), the proof would finally check \(u = u_{prev} + \delta\). if we want a "three strikes and you're out" rule, the zk-snark would also check \(u > -3\). if we want a rule where a post can get a special "high-reputation poster" flag if the poster has \(\ge 100\) rep, we can accommodate that by adding "is \(u \ge 100\)?" as a public input. many kinds of such rules can be accommodated. to increase the scalability of the scheme, we could split it up into two kinds of messages: posts and reputation update acknowledgements (rcas). a post would be off-chain, though it would be required to point to an rca made in the past week. rcas would be on-chain, and an rca would walk through all the reputation updates since that poster's previous rca. this way, the on-chain load is reduced to one transaction per poster per week plus one transaction per reputation message (a very low level if reputation updates are rare, eg. they're only used for moderation actions or perhaps "post of the day" style prizes). holding centralized parties accountable sometimes, you need to build a scheme that has a central "operator" of some kind. this could be for many reasons: sometimes it's for scalability, and sometimes it's for privacy specifically, the privacy of data held by the operator. the maci coercion-resistant voting system, for example, requires voters to submit their votes on-chain encrypted to a secret key held by a central operator. the operator would decrypt all the votes on-chain, count them up, and reveal the final result, along with a zk-snark proving that they did everything correctly. this extra complexity is necessary to ensure a strong privacy property (called coercion-resistance): that users cannot prove to others how they voted even if they wanted to. thanks to blockchains and zk-snarks, the amount of trust in the operator can be kept very low. a malicious operator could still break coercion resistance, but because votes are published on the blockchain, the operator cannot cheat by censoring votes, and because the operator must provide a zk-snark, they cannot cheat by mis-calculating the result. combining zk-snarks with mpc a more advanced use of zk-snarks involves making proofs over computations where the inputs are split between two or more parties, and we don't want each party to learn the other parties' inputs. you can satisfy the privacy requirement with garbled circuits in the 2-party case, and more complicated multi-party computation protocols in the n-party case. zk-snarks can be combined with these protocols to do verifiable multi-party computation. this could enable more advanced reputation systems where multiple participants can perform joint computations over their private inputs, it could enable privacy-preserving but authenticated data markets, and many other applications. that said, note that the math for doing this efficiently is still relatively in its infancy. what can't we make private? zk-snarks are generally very effective for creating systems where users have private state. but zk-snarks cannot hold private state that nobody knows. to make a proof about a piece of information, the prover has to know that piece of information in cleartext. a simple example of what can't (easily) be made private is uniswap. in uniswap, there is a single logically-central "thing", the market maker account, which belongs to no one, and every single trade on uniswap is trading against the market maker account. you can't hide the state of the market maker account, because then someone would have to hold the state in cleartext to make proofs, and their active involvement would be necessary in every single transaction. you could make a centrally-operated, but safe and private, uniswap with zk-snarked garbled circuits, but it's not clear that the benefits of doing this are worth the costs. there may not even be any real benefit: the contract would need to be able to tell users what the prices of the assets are, and the block-by-block changes in the prices tell a lot about what the trading activity is. blockchains can make state information global, zk-snarks can make state information private, but we don't really have any good way to make state information global and private at the same time. edit: you can use multi-party computation to implement shared private state. but this requires an honest-majority threshold assumption, and one that's likely unstable in practice because (unlike eg. with 51% attacks) a malicious majority could collude to break the privacy without ever being detected. putting the primitives together in the sections above, we've seen some examples that are powerful and useful tools by themselves, but they are also intended to serve as building blocks in other applications. nullifiers, for example, are important for currency, but it turns out that they pop up again and again in all kinds of use cases. the "forced chaining" technique used in the negative reputation section is very broadly applicable. it's effective for many applications where users have complex "profiles" that change in complex ways over time, and you want to force the users to follow the rules of the system while preserving privacy so no one sees which user is performing which action. users could even be required to have entire private merkle trees representing their internal "state". the "commitment pool" gadget proposed in this post could be built with zk-snarks. and if some application can't be entirely on-chain and must have a centralized operator, the exact same techniques can be used to keep the operator honest too. zk-snarks are a really powerful tool for combining together the benefits of accountability and privacy. they do have their limits, though in some cases clever application design can work around those limits. i expect to see many more applications using zk-snarks, and eventually applications combining zk-snarks with other forms of cryptography, to be built in the years to come. on transaction fees, and the fallacy of market-based solutions | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on transaction fees, and the fallacy of market-based solutions posted by vitalik buterin on february 1, 2014 research & development of all the parts of the ethereum protocol, aside from the mining function the fee structure is perhaps the least set in stone. the current values, with one crypto operation taking 20 base fees, a new transaction taking 100 base fees, etc, are little more than semi-educated guesses, and harder data on exactly how much computational power a database read, an arithmetic operation and a hash actually take will certainly give us much better estimates on what exactly the ratios between the different computational fees should be. the other part of the question, that of exactly how much the base fee should be, is even more difficult to figure out; we have still not decided whether we want to target a certain block size, a certain usd-denominated level, or some combination of these factors, and it is very difficulty to say whether a base fee of 0.00001orabasefeeof0.00001 or a base fee of 0.00001orabasefeeof0.001 would be more appropriate. ultimately, what is becoming more and more clear to us is that some kind of flexible fee system, that allows consensus-based human intervention after the fact, would be best for the project. when many people coming from bitcoin see this problem, however, they wonder why we are having such a hard time with this issue when bitcoin already has a ready-made solution: make the fees voluntary and market-based. in the bitcoin protocol, there are no mandatory transaction fees; even an extremely large and computationally arduous transaction can get in with a zero fee, and it is up to the miners to determine what fees they require. the lower a transaction’s fee, the longer it takes for the transaction to find a miner that will let it in, and those who want faster confirmations can pay more. at some point, an equilibrium should be reached. problem solved. so why not here? the reality, is, however, is that in bitcoin the transaction fee problem is very far from “solved”. the system as described above already has a serious vulnerability: miners have to pay no fees, so a miner can choke the entire network with an extremely large block. in fact, this problem is so serious that satoshi close to fix it with the ugliest possible path: set a maximum block size limit of 1 mb, or 7 transactions per second. now, without the immensely hard-fought and politically laden debate that necessarily accompanies any “hard-forking” protocol change, bitcoin simply cannot organically adapt to handle anything more than the 7 tx/sec limit that satoshi originally placed. and that’s bitcoin. in ethereum, the issue is even more problematic due to turing-completeness. in bitcoin, one can construct a mathematical proof that a transaction n bytes long will not take more than k*n time to verify for some constant k. in ethereum, one can construct a transaction in less than 150 bytes that, absent fees, will run forever: [ to, value, [ push, 0, jmp ], v, r, s ] in case you do not understand that, it’s the equivalent of 10: do_nothing, 20: goto 10; an infinite loop. and as soon as a miner publishes a block that includes that transaction, the entire network will freeze. in fact, thanks to the well-known impossibility of the halting problem, it is not even possible to construct a filter to weed out infinite-looping scripts. thus, computational attacks on ethereum are trivial, and even more restrictions must be placed in order to ensure that ethereum remains a workable platform. but wait, you might say, why not just take the 1 mb limit, and convert it into a 1 million x base fee limit? one can even make the system more future-proof by replacing a hard cap with a floating cap of 100 times the moving average of the last 10000 blocks. at this point, we need to get deeper into the economics and try to understand what “market-based fees” are all about. crypto, meet pigou in general terms, an idealized market, or at least one specific subset of a market, can be defined as follows. there exist a set of sellers, s[1] ... s[n], who are interested in selling a particular resource, and where seller s[i] incurs a cost c[i] from giving up that resource. we can say c[1] < c[2] < ... < c[n] for simplicity. similarly, there exist some buyers, b[1] ... b[n], who are interested in gaining a particular resource and incur a gain g[i], where g[1] > g[2] > ... > g[n]. then, an order matching process happens as follows. first, one locates the last k where g[k] > c[k]. then, one picks a price between those two values, say at p = (g[k] + c[k])/2, and s[i] and b[i] make a trade, where s[i] gives the resource to b[i] and b[i] pays p to s[i]. all parties benefit, and the benefit is the maximum possible; if s[k+1] and b[k+1] also made a transaction, c[k+1] > v[k+1], so the transaction would actually have negative net value to society. fortunately, it is in everybody’s interest to make sure that they do not participate in unfavorable trades. the question is, is this kind of market the right model for bitcoin transactions? to answer this question, let us try to put all of the players into roles. the resource is the service of transaction processing, and the people benefitting from the resource, the transaction senders, are also the buyers paying transaction fees. so far, so good. the sellers are obvious the miners. but who is incurring the costs? here, things get tricky. for each individual transaction that a miner includes, the costs are borne not just by that miner, but by every single node in the entire network. the cost per transaction is tiny; a miner can process a transaction and include it in a block for less than 0.00001worthofelectricityanddatastorage.thereasonwhytransactionfeesneedtobehighisbecausethat0.00001 worth of electricity and data storage. the reason why transaction fees need to be high is because that 0.00001worthofelectricityanddatastorage.thereasonwhytransactionfeesneedtobehighisbecausethat0.00001 is being paid by thousands of nodes all around the world. it gets worse. suppose that the net cost to the network of processing a transaction is close to 0.05.intheory,evenifthecostsarenotbornebyexactlythesamepeoplewhosettheprices,aslongasthetransactionfeeiscloseto0.05. in theory, even if the costs are not borne by exactly the same people who set the prices, as long as the transaction fee is close to 0.05.intheory,evenifthecostsarenotbornebyexactlythesamepeoplewhosettheprices,aslongasthetransactionfeeiscloseto0.05 the system would still be in balance. but what is the equilibrium transaction fee going to be? right now, fees are around 0.09simplybecauseminersaretoolazytoswitch.butthen,inthefuture,whathappensoncefeesbecomealargershareofaminer’srevenueandminershavealargeincentivetotrytomaximizetheirtake?theobviousansweris,forasolominertheequilibriumtransactionfeeis0.09 simply because miners are too lazy to switch. but then, in the future, what happens once fees become a larger share of a miner’s revenue and miners have a large incentive to try to maximize their take? the obvious answer is, for a solo miner the equilibrium transaction fee is 0.09simplybecauseminersaretoolazytoswitch.butthen,inthefuture,whathappensoncefeesbecomealargershareofaminer’srevenueandminershavealargeincentivetotrytomaximizetheirtake?theobviousansweris,forasolominertheequilibriumtransactionfeeis0.00001. if a transaction with a fee of 0.00002comesin,andthemineraddsit,theminerwillhaveearnedaprofitof0.00002 comes in, and the miner adds it, the miner will have earned a profit of 0.00002comesin,andthemineraddsit,theminerwillhaveearnedaprofitof0.00001, and the remaining $0.04999 worth of costs will be paid by the rest of the network together – a cryptographic tragedy of the commons. now, suppose that the mining ecosystem is more oligarchic, with one pool controlling 25% of all mining power. what are the incentives then? here, it gets more tricky. the mining pool can actually choose to set its minimum fee higher, perhaps at 0.001.thismayseemlikethepoolisforgoingprofitopportunitiesbetween0.001. this may seem like the pool is forgoing profit opportunities between 0.001.thismayseemlikethepoolisforgoingprofitopportunitiesbetween0.00001 and 0.00099,butwhatisalsohappeningisthatmanyofthetransactionsenderswhowereaddingbetween0.00099, but what is also happening is that many of the transaction senders who were adding between 0.00099,butwhatisalsohappeningisthatmanyofthetransactionsenderswhowereaddingbetween0.00001 and $0.00099 before now have the incentive to increase their fees to make sure this pool confirms their transactions – otherwise, they would need to wait an average of 3.3 minutes longer. thus, the fewer miners there are, the higher fees go – even thought a reduced number of miners actually means a lower network cost of processing all transactions. from the above discussion, what should become painfully clear is that transaction processing simply is not a market, and therefore trying to apply market-like mechanisms to it is an exercise in random guessing at best, and a scalability disaster at worst. so what are the alternatives? the economically ideal solution is one that has often been brought up in the context of global warming, perhaps the largest geopolitical tragedy of the commons scenario in the modern world: pigovian taxes. price setting without a market the way a pigovian tax works is simple. through some mechanism, the total net cost of consuming a certain quantity of a common resource (eg. network computation, air purity) is calculated. then, everyone who consumes that resource is required to pay that cost for every unit of the resource that they consume (or for every unit of pollution that they emit). the challenge in pigovian taxation, however, is twofold. first, who gets the revenue? second, and more importantly, there is no way to opt out of pollution, and thus no way for the market to extract people’s preferences about how much they would need to gain in order to suffer a given dose of pollution; thus, how do we set the price? in general, there are three ways of solving this problem: philosopher kings set the price, and disappear as the price is set in stone forever. philosopher kings maintain active control over the price. some kind of democratic mechanism there is also a fourth way, some kind of market mechanism which randomly doles out extra pollution to certain groups and attempts to measure the extent to which people (or network nodes in the context of a crytocurrency) are willing to go to avoid that pollution; this approach is interesting but heavily underexplored, and i will not attempt to examine it at this point in time. our initial strategy was (1). ripple’s strategy is (2). now, we are increasingly looking to (3). but how would (3) be implemented? fortunately, cryptocurrency is all about democratic consensus, and every cryptocurrency already has at least two forms of consensus baked in: proof of work and proof of stake. i will show two very simple protocols for doing this right now: proof of work protocol if you mine a block, you have the right to set a value in the “extra data field”, which can be anywhere from 0-32 bytes (this is already in the protocol) if the first byte of this data is 0, nothing happens if the first byte of this data is 1, we set block.basefee = block.basefee + floor(block.basefee / 65536) if the first byte of this data is 255, we set block.basefee = block.basefee floor(block.basefee / 65536) proof of stake protocol after each block, calculate h = sha256(block.parenthash + address) * block.address_balance(address)for each address if h > 2^256 / difficulty, where difficulty is a set constant, that address can sign either 1, 0 or 255 and create a signed object of the form [ val, v, r, s ] the miner can then include that object in the block header, giving the miner and the stakeholder some miniscule reward. if the data is 1, we set block.basefee = block.basefee + floor(block.basefee / 65536) if the data is 255, we set block.basefee = block.basefee floor(block.basefee / 65536) the two protocols are functionally close to identical; the only difference is that in the proof of work protocol miners decide on the basefee and in the proof of stake protocol ether holders do. the question is, do miners and ether holders have their incentives aligned to set the fee fairly? if transaction fees go to miners, then miners clearly do not. however, if transaction fees are burned, and thus their value goes to all ether holder proportionately through reduced inflation, then perhaps they do. miners and ether holders both want to see the value of their ether go up, so they want to set a fee that makes the network more useful, both in terms of not making it prohibitively expensive to make transactions and in terms of not setting a high computational load. thus, in theory, assuming rational actors, we will have fees that are at least somewhat reasonable. is there a reason to go one way or the other in terms of miners versus ether holders? perhaps there is. miners have the incentive to see the value of ether go as high as possible in the short term, but perhaps not so much in the long term, since a prolonged rise eventually brings competition which cancels out the miners’ increased profit. thus, miners might end up adopting a looser policy that imposes higher costs (eg. data storage) on miners far down the line. ether holders, on the other hand, seem to have a longer term interest. on the other hand, miners are somewhat “locked in” to mining ether specifically, especially if semi-specialized or specialized hardware gets involved; ether holders, on the other hand, can easily hop from one market to the other. furthermore, miners are less anonymous than ether holders. thus, the issue is not clear cut; if transaction fees are burned one can go either way. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle alternatives to selling at below-market-clearing prices for achieving fairness (or community sentiment, or fun) 2021 aug 22 see all posts when a seller wants to sell a fixed supply of an item that is in high (or uncertain and possibly high) demand, one choice that they often make is to set a price significantly lower than what "the market will bear". the result is that the item quickly sells out, with the lucky buyers being those who attempted to buy first. this has happened in a number of situations within the ethereum ecosystem, notably nft sales and token sales / icos. but this phenomenon is much older than that; concerts and restaurants frequently make similar choices, keeping prices cheap and leading to seats quickly selling out or buyers waiting in long lines. economists have for a long time asked the question: why do sellers do this? basic economic theory suggests that it's best if sellers sell at the market-clearing price that is, the price at which the amount that buyers are willing to buy exactly equals the amount the seller has to sell. if the seller doesn't know what the market-clearing price is, the seller should sell through an auction, and let the market determine the price. selling below market-clearing price not only sacrifices revenue for the seller; it also can harm the buyers: the item may sell out so quickly that many buyers have no opportunity to get it at all, no matter how much they want it and are willing to pay to get it. sometimes, the competitions created by these non-price-based allocation mechanisms even create negative externalities that harm third parties an effect that, as we will see, is particularly severe in the ethereum ecosystem. but nevertheless, the fact that below-market-clearing pricing is so prevalent suggests that there must be some convincing reasons why sellers do it. and indeed, as the research into this topic over the last few decades has shown, there often are. and so it's worth asking the question: are there ways of achieving the same goals with more fairness, less inefficiency and less harm? selling at below market-clearing prices has large inefficiencies and negative externalities if a seller sells an item at market price, or through an auction, someone who really really wants that item has a simple path to getting it: they can pay the high price or if it's an auction they can bid a high amount. if a seller sells the item at below market price, then demand exceeds supply, and so some people will get the item and others won't. but the mechanism deciding who will get the item is decidedly not random, and it's often not well-correlated with how much participants want the item. sometimes, it involves being faster at clicking buttons than everyone else. at other times, it involves waking up at 2 am in your timezone (but 11 pm or even 2 pm in someone else's). and at still other times, it just turns into an "auction by other means", one which is more chaotic, less efficient and laden with far more negative externalties. within the ethereum ecosystem, there are many clear examples of this. first, we can look at the ico craze of 2017. in 2017, there were a large number of projects launching initial coin offerings (icos), and a typical model was the capped sale: the project would set the price of the token and a hard maximum for how many tokens they are willing to sell, and at some point in time the sale would start automatically. once the number of tokens hit the cap, the sale ends. what's the result? in practice, these sales would often end in as little as 30 seconds. as soon as (or rather, just before) the sale starts, everyone would start sending transactions in to try to get in, offering higher and higher fees to encourage miners to include their transaction first. an auction by another name except with revenues going to the miners instead of the token seller, and the extremely harmful negative externality of pricing out every other application on-chain while the sale is going on. the most expensive transaction in the bat sale set a fee of 580,000 gwei, paying a fee of $6,600 to get included in the sale. many icos after that tried various strategies to avoid these gas price auctions; one ico notably had a smart contract that checked the transaction's gasprice and rejected it if it exceeded 50 gwei. but that of course, did not solve the problem. buyers wishing to cheat the system sent many transactions, hoping that at least one would get in. once again, an auction by another name, and this time clogging up the chain even more. in more recent times, icos have become less popular, but nfts and nft sales are now very popular. unfortunately, the nft space failed to learn the lessons from 2017; they make fixed-quantity fixed-supply sales just like the icos did (eg. see the mint function on lines 97-108 of this contract here). what's the result? and this isn't even the biggest one; some nft sales have created gas price spikes as high as 2000 gwei. once again, sky-high gas prices from users fighting each other by sending higher and higher transaction fees to get in first. an auction by another name, pricing out every other application on-chain for 15 minutes, just as before. so why do sellers sometimes sell below market price? selling at below market price is hardly a new phenomenon, both within the blockchain space and outside, and over the decades there have been many articles and papers and podcasts writing (and sometimes bitterly complaining) about the unwillingness to use auctions or set prices to market-clearing levels. many of the arguments are very similar between the examples in the blockchain space (nfts and icos) and outside the blockchain space (popular restaurants and concerts). a particular concern is fairness and the desire to not lock poorer people out and not lose fans or create tension as a result of being perceived as greedy. kahneman, knetsch and thaler's 1986 paper is a good exposition of how perceptions of fairness and greed can influence these decisions. in my own recollection of the 2017 ico season, the desire to avoid perceptions of greed was similarly a decisive factor in discouraging the use of auction-like mechanisms (i am mostly going off memory here and do not have many sources, though i did find a link to a no-longer-available parody video making some kind of comparison between the auction-based gnosis ico and the national socialist german workers' party). in addition to fairness issues, there are also the perennial arguments that products selling out and having long lines creates a perception of popularity and prestige, which makes the product seem even more attractive to others further down the line. sure, in a rational actor model, high prices should have the same effect as long lines, but in reality long lines are much more visible than high prices are. this is just as true for icos and nfts as it is for restaurants. in addition to these strategies generating more marketing value, some people actually find participating in or watching the game of grabbing up a limited set of opportunities first before everyone else takes them all to be quite fun. but there are also some factors specific to the blockchain space. one argument for selling ico tokens at below-market-clearing prices (and one that was decisive in convincing the omisego team to adopt their capped sale strategy) has to do with community dynamics of token issuance. the most basic rule of community sentiment management is simple: you want prices to go up, not down. if community members are "in the green", they are happy. but if the price goes lower than what it was when the community members bought, leaving them at a net loss, they become unhappy and start calling you a scammer, and possibly creating a social media cascade leading to everyone else calling you a scammer. the only way to avoid this effect is to set a sale price low enough that the post-launch market price will almost certainly be higher. but, how do you actually do this without creating a rush-for-the-gates dynamic that leads to an auction by other means? some more interesting solutions the year is 2021. we have a blockchain. the blockchain contains not just a powerful decentralized finance ecosystem, but also a rapidly growing suite of all kinds of non-financial tools. the blockchain also presents us with a unique opportunity to reset social norms. uber legitimized surge pricing where decades of economists yelling about "efficiency" failed; surely, blockchains can also be an opportunity to legitimize new uses of mechanism design. and surely, instead of fiddling around with a coarse-grained one-dimensional strategy space of selling at market price versus below market price (with perhaps a second dimension for auction versus fixed-price sale), we could use our more advanced tools to create an approach that more directly solves the problems, with fewer side effects? first, let us list the goals. we'll try to cover the cases of (i) icos, (ii) nfts and (iii) conference tickets (really a type of nft) at the same time; most of the desired properties are shared between the three cases. fairness: don't completely lock low-income people out of participating, give them at least some chance to get in. for token sales, there's the not quite identical but related goal of avoiding high initial wealth concentration and having a larger and more diverse initial token holder community. don't create races: avoid creating situations where lots of people are rushing to take the same action and only the first few get in (this is the type of situation that leads to the horrible auctions-by-another-name that we saw above). don't require fine-grained knowledge of market conditions: the mechanism should work even if the seller has absolutely no idea how much demand there is. fun: the process of participating in the sale should ideally be interesting and have game-like qualities, but without being frustrating. give buyers positive expected returns: in the case of a token (or, for that matter, an nft), buyers should be more likely to see the item go up in price than go down. this necessarily implies selling to buyers at below the market price. we can start by looking at (1). looking at it from the point of view of ethereum, there is a pretty clear solution. instead of creating race conditions, just use an explicitly designed tool for the job: proof of personhood protocols! here's one quick proposed mechanism: mechanism 1 each participant (verified by proof-of-personhood) can buy up to x units at price p, and if they want to buy more they can buy in an auction. it seems like it satisfies a lot of the goals already: the per-person aspect provides fairness, if the auction price turns out higher than p buyers can get positive expected returns for the portion sold through the per-person mechanism, and the auction part does not require the seller to understand the level of demand. does it avoid creating races? if the number of participants buying through the per-person pool is not that high, it seems like it does. but what if so many people show up that the per-person pool is not big enough to provide an allocation for all of them? here's an idea: make the per-person allocation amount itself dynamic. mechanism 2 each participant (verified by proof-of-personhood) can make a deposit into a smart contract to declare interest for up to x tokens. at the end, each buyer is given an allocation of min(x, n / number_of_buyers) tokens, where n is the total amount sold through the per-person pool (some other amount can also be sold by auction). the portion of the buyer's deposit going above the amount needed to buy their allocation is refunded to them. now, there's no race condition regardless of the number of buyers going through the per-person pool. no matter how high the demand, there's no way in which it's more beneficial to participate earlier rather than later. here's yet another idea, if you like your game mechanics to be more clever and use fancy quadratic formulas. mechanism 3 each participant (verified by proof-of-personhood) can buy \(x\) units at a price \(p * x^2\), up to a maximum of \(c\) tokens per buyer. \(c\) starts at some low number, and then increases over time until enough units are sold. this mechanism has the particularly interesting property that if you're making a governance token (please don't do that; this is purely harm-reduction advice), the quantity allocated to each buyer is theoretically optimal, though of course post-sale transfers will degrade this optimality over time. mechanisms 2 and 3 seem like they both satisfy all of the above goals, at least to some extent. they're not necessarily perfect and ideal, but they do make good starting points. there is one remaining issue. for fixed and limited-supply nfts, you might get the problem that the equilibrium purchased quantity per participant is fractional (in mechanism 2, perhaps number_of_buyers > n, and in mechanism 3, perhaps setting \(c = 1\) already leads to enough demand to over-subscribe the sale). in this case, you can sell fractional items by offering lottery tickets: if there are n items to be sold, then if you subscribe you have a chance of n / number_of_buyers that you will actually get the item, and otherwise you get a refund. for a conference, groups that want to go together could be allowed to bundle their lottery tickets to guarantee either all-win or all-lose. ability to get the item for certain can be sold at auction. a fun mildly-grey-hat tactic for conference tickets is to disguise the pool being sold at market rate as the bottom tier of "sponsorships". you may end up with a bunch of people's faces on the sponsor board, but... maybe that's fine? after all, ethcc had john lilic's face on their sponsor board! in all of these cases, the core of the solution is simple: if you want to be reliably fair to people, then your mechanism should have some input that explicitly measures people. proof of personhood protocols do this (and if desired can be combined with zero knowledge proofs to ensure privacy). ergo, we should take the efficiency benefits of market and auction-based pricing, and the egalitarian benefits of proof of personhood mechanics, and combine them together. answers to possible questions q: wouldn't lots of people who don't even care about your project buy the item through the egalitarian scheme and immediately resell it? a: initially, probably not. in practice, such meta-games take time to show up. but if/when they do, one possible mitigation is to make them untradeable for some period of time. this actually works because proof-of-personhood identities are untradeable: you can always use your face to claim that your previous account got hacked and the identity corresponding to you, including everything in it, should be moved to a new account. q: what if i want to make my item accessible not just to people in general, but to a particular community? a: instead of proof of personhood, use proof of participation tokens connected to events in that community. an additional alternative, also serving both egalitarian and gamification value, is to lock some items inside solutions to some publicly-published puzzles. q: how do we know people will accept this? people have been resistant to weird new mechanisms in the past. a: it's very difficult to get people to accept a new mechanism that they find weird by having economists write screeds about how they "should" accept it for the sake of "efficiency" (or even "equity"). however, rapid changes in context do an excellent job of resetting people's set expectations. so if there's any good time at all to try this, the blockchain space is that time. you could also wait for the "metaverse", but it's quite possible that the best version of the metaverse will run on ethereum anyway, so you might as well just start now. dark mode toggle two thought experiments to evaluate automated stablecoins 2022 may 25 see all posts special thanks to dan robinson, hayden adams and dankrad feist for feedback and review. the recent luna crash, which led to tens of billions of dollars of losses, has led to a storm of criticism of "algorithmic stablecoins" as a category, with many considering them to be a "fundamentally flawed product". the greater level of scrutiny on defi financial mechanisms, especially those that try very hard to optimize for "capital efficiency", is highly welcome. the greater acknowledgement that present performance is no guarantee of future returns (or even future lack-of-total-collapse) is even more welcome. where the sentiment goes very wrong, however, is in painting all automated pure-crypto stablecoins with the same brush, and dismissing the entire category. while there are plenty of automated stablecoin designs that are fundamentally flawed and doomed to collapse eventually, and plenty more that can survive theoretically but are highly risky, there are also many stablecoins that are highly robust in theory, and have survived extreme tests of crypto market conditions in practice. hence, what we need is not stablecoin boosterism or stablecoin doomerism, but rather a return to principles-based thinking. so what are some good principles for evaluating whether or not a particular automated stablecoin is a truly stable one? for me, the test that i start from is asking how the stablecoin responds to two thought experiments. click here to skip straight to the thought experiments. reminder: what is an automated stablecoin? for the purposes of this post, an automated stablecoin is a system that has the following properties: it issues a stablecoin, which attempts to target a particular price index. usually, the target is 1 usd, but there are other options too. there is some targeting mechanism that continuously works to push the price toward the index if it veers away in either direction. this makes eth and btc not stablecoins (duh). the targeting mechanism is completely decentralized, and free of protocol-level dependencies on specific trusted actors. particularly, it must not rely on asset custodians. this makes usdt and usdc not automated stablecoins. in practice, (2) means that the targeting mechanism must be some kind of smart contract which manages some reserve of crypto-assets, and uses those crypto-assets to prop up the price if it drops. how does terra work? terra-style stablecoins (roughly the same family as seignorage shares, though many implementation details differ) work by having a pair of two coins, which we'll call a stablecoin and a volatile-coin or volcoin (in terra, ust is the stablecoin and luna is the volcoin). the stablecoin retains stability using a simple mechanism: if the price of the stablecoin exceeds the target, the system auctions off new stablecoins (and uses the revenue to burn volcoins) until the price returns to the target if the price of the stablecoin drops below the target, the system buys back and burns stablecoins (issuing new volcoins to fund the burn) until the price returns to the target now what is the price of the volcoin? the volcoin's value could be purely speculative, backed by an assumption of greater stablecoin demand in the future (which would require burning volcoins to issue). alternatively, the value could come from fees: either trading fees on stablecoin <-> volcoin exchange, or holding fees charged per year to stablecoin holders, or both. but in all cases, the price of the volcoin comes from the expectation of future activity in the system. how does rai work? in this post i'm focusing on rai rather than dai because rai better exemplifies the pure "ideal type" of a collateralized automated stablecoin, backed by eth only. dai is a hybrid system backed by both centralized and decentralized collateral, which is a reasonable choice for their product but it does make analysis trickier. in rai, there are two main categories of participants (there's also holders of flx, the speculative token, but they play a less important role): a rai holder holds rai, the stablecoin of the rai system. a rai lender deposits some eth into a smart contract object called a "safe". they can then withdraw rai up to the value of \(\frac{2}{3}\) of that eth (eg. if 1 eth = 100 rai, then if you deposit 10 eth you can withdraw up to \(10 * 100 * \frac{2}{3} \approx 667\) rai). a lender can recover the eth in the same if they pay back their rai debt. there are two main reasons to become a rai lender: to go long on eth: if you deposit 10 eth and withdraw 500 rai in the above example, you end up with a position worth 500 rai but with 10 eth of exposure, so it goes up/down by 2% for every 1% change in the eth price. arbitrage if you find a fiat-denominated investment that goes up faster than rai, you can borrow rai, put the funds into that investment, and earn a profit on the difference. if the eth price drops, and a safe no longer has enough collateral (meaning, the rai debt is now more than \(\frac{2}{3}\) times the value of the eth deposited), a liquidation event takes place. the safe gets auctioned off for anyone else to buy by putting up more collateral. the other main mechanism to understand is redemption rate adjustment. in rai, the target isn't a fixed quantity of usd; instead, it moves up or down, and the rate at which it moves up or down adjusts in response to market conditions: if the price of rai is above the target, the redemption rate decreases, reducing the incentive to hold rai and increasing the incentive to hold negative rai by being a lender. this pushes the price back down. if the price of rai is below the target, the redemption rate increases, increasing the incentive to hold rai and reducing the incentive to hold negative rai by being a lender. this pushes the price back up. thought experiment 1: can the stablecoin, even in theory, safely "wind down" to zero users? in the non-crypto real world, nothing lasts forever. companies shut down all the time, either because they never manage to find enough users in the first place, or because once-strong demand for their product is no longer there, or because they get displaced by a superior competitor. sometimes, there are partial collapses, declines from mainstream status to niche status (eg. myspace). such things have to happen to make room for new products. but in the non-crypto world, when a product shuts down or declines, customers generally don't get hurt all that much. there are certainly some cases of people falling through the cracks, but on the whole shutdowns are orderly and the problem is manageable. but what about automated stablecoins? what happens if we look at a stablecoin from the bold and radical perspective that the system's ability to avoid collapsing and losing huge amounts of user funds should not depend on a constant influx of new users? let's see and find out! can terra wind down? in terra, the price of the volcoin (luna) comes from the expectation of fees from future activity in the system. so what happens if expected future activity drops to near-zero? the market cap of the volcoin drops until it becomes quite small relative to the stablecoin. at that point, the system becomes extremely fragile: only a small downward shock to demand for the stablecoin could lead to the targeting mechanism printing lots of volcoins, which causes the volcoin to hyperinflate, at which point the stablecoin too loses its value. the system's collapse can even become a self-fulfilling prophecy: if it seems like a collapse is likely, this reduces the expectation of future fees that is the basis of the value of the volcoin, pushing the volcoin's market cap down, making the system even more fragile and potentially triggering that very collapse exactly as we saw happen with terra in may. luna price, may 8-12 ust price, may 8-12 first, the volcoin price drops. then, the stablecoin starts to shake. the system attempts to shore up stablecoin demand by issuing more volcoins. with confidence in the system low, there are few buyers, so the volcoin price rapidly falls. finally, once the volcoin price is near-zero, the stablecoin too collapses. in principle, if demand decreases extremely slowly, the volcoin's expected future fees and hence its market cap could still be large relative to the stablecoin, and so the system would continue to be stable at every step of its decline. but this kind of successful slowly-decreasing managed decline is very unlikely. what's more likely is a rapid drop in interest followed by a bang. safe wind-down: at every step, there's enough expected future revenue to justify enough volcoin market cap to keep the stablecoin safe at its current level. unsafe wind-down: at some point, there's not enough expected future revenue to justify enough volcoin market cap to keep the stablecoin safe. collapse is likely. can rai wind down? rai's security depends on an asset external to the rai system (eth), so rai has a much easier time safely winding down. if the decline in demand is unbalanced (so, either demand for holding drops faster or demand for lending drops faster), the redemption rate will adjust to equalize the two. the lenders are holding a leveraged position in eth, not flx, so there's no risk of a positive-feedback loop where reduced confidence in rai causes demand for lending to also decrease. if, in the extreme case, all demand for holding rai disappears simultaneously except for one holder, the redemption rate would skyrocket until eventually every lender's safe gets liquidated. the single remaining holder would be able to buy the safe in the liquidation auction, use their rai to immediately clear its debt, and withdraw the eth. this gives them the opportunity to get a fair price for their rai, paid for out of the eth in the safe. another extreme case worth examining is where rai becomes the primary appliation on ethereum. in this case, a reduction in expected future demand for rai would crater the price of eth. in the extreme case, a cascade of liquidations is possible, leading to a messy collapse of the system. but rai is far more robust against this possibility than a terra-style system. thought experiment 2: what happens if you try to peg the stablecoin to an index that goes up 20% per year? currently, stablecoins tend to be pegged to the us dollar. rai stands out as a slight exception, because its peg adjusts up or down due to the redemption rate and the peg started at 3.14 usd instead of 1 usd (the exact starting value was a concession to being normie-friendly, as a true math nerd would have chosen tau = 6.28 usd instead). but they do not have to be. you can have a stablecoin pegged to a basket of assets, a consumer price index, or some arbitrarily complex formula ("a quantity of value sufficient to buy {global average co2 concentration minus 375} hectares of land in the forests of yakutia"). as long as you can find an oracle to prove the index, and people to participate on all sides of the market, you can make such a stablecoin work. as a thought experiment to evaluate sustainability, let's imagine a stablecoin with a particular index: a quantity of us dollars that grows by 20% per year. in math language, the index is \(1.2^{(t t_0)}\) usd, where \(t\) is the current time in years and \(t_0\) is the time when the system launched. an even more fun alternative is \(1.04^{\frac{1}{2}*(t t_0)^2}\) usd, so it starts off acting like a regular usd-denominated stablecoin, but the usd-denominated return rate keeps increasing by 4% every year. obviously, there is no genuine investment that can get anywhere close to 20% returns per year, and there is definitely no genuine investment that can keep increasing its return rate by 4% per year forever. but what happens if you try? i will claim that there's basically two ways for a stablecoin that tries to track such an index to turn out: it charges some kind of negative interest rate on holders that equilibrates to basically cancel out the usd-denominated growth rate built in to the index. it turns into a ponzi, giving stablecoin holders amazing returns for some time until one day it suddenly collapses with a bang. it should be pretty easy to understand why rai does (1) and luna does (2), and so rai is better than luna. but this also shows a deeper and more important fact about stablecoins: for a collateralized automated stablecoin to be sustainable, it has to somehow contain the possibility of implementing a negative interest rate. a version of rai programmatically prevented from implementing negative interest rates (which is what the earlier single-collateral dai basically was) would also turn into a ponzi if tethered to a rapidly-appreciating price index. even outside of crazy hypotheticals where you build a stablecoin to track a ponzi index, the stablecoin must somehow be able to respond to situations where even at a zero interest rate, demand for holding exceeds demand for borrowing. if you don't, the price rises above the peg, and the stablecoin becomes vulnerable to price movements in both directions that are quite unpredictable. negative interest rates can be done in two ways: rai-style, having a floating target that can drop over time if the redemption rate is negative actually having balances decrease over time option (1) has the user-experience flaw that the stablecoin no longer cleanly tracks "1 usd". option (2) has the developer-experience flaw that developers aren't used to dealing with assets where receiving n coins does not unconditionally mean that you can later send n coins. but choosing one of the two seems unavoidable unless you go the makerdao route of being a hybrid stablecoin that uses both pure cryptoassets and centralized assets like usdc as collateral. what can we learn? in general, the crypto space needs to move away from the attitude that it's okay to achieve safety by relying on endless growth. it's certainly not acceptable to maintain that attitude by saying that "the fiat world works in the same way", because the fiat world is not attempting to offer anyone returns that go up much faster than the regular economy, outside of isolated cases that certainly should be criticized with the same ferocity. instead, while we certainly should hope for growth, we should evaluate how safe systems are by looking at their steady state, and even the pessimistic state of how they would fare under extreme conditions and ultimately whether or not they can safely wind down. if a system passes this test, that does not mean it's safe; it could still be fragile for other reasons (eg. insufficient collateral ratios), or have bugs or governance vulnerabilities. but steady-state and extreme-case soundness should always be one of the first things that we check for. a message from stephan tual | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search a message from stephan tual posted by stephan tual on september 3, 2015 organizational to the wonderful ethereum community, you often heard me say at conferences that ethereum was not a company, a foundation, an implementation, or an individual. ethereum is both an idea and an ideal, encompassing the first censorship-resistant network build specifically to enable those who need it the most to safely trade, privately self-organise and freely communicate, rather than relying on the crippled walled garden handed out by the powers that be. due to divergence in personal values, eth/dev and i have mutually decided to part ways. i of course intend to continue promoting the ethereum ideals and bring about a world-class, mainstream platform to build a new breed of decentralized applications. from now on, you'll be able to email me at stephan@ursium.com, read my articles and tutorials on blog.ursium.com, or follow me at @stephantual. i want to thank each and everyone of you for your incredible enthusiasm and passion, helping pioneer what i believe is the most significant piece of software concept the world has ever seen. your commitment inspired me, and on many occasions helped me keep going through a year and half of incredibly hard work. see you very soon, stephan previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle convex and concave dispositions 2020 nov 08 see all posts one of the major philosophical differences that i have noticed in how people approach making large-scale decisions in the world is how they approach the age-old tradeoff of compromise versus purity. given a choice between two alternatives, often both expressed as deep principled philosophies, do you naturally gravitate toward the idea that one of the two paths should be correct and we should stick to it, or do you prefer to find a way in the middle between the two extremes? in mathematical terms, we can rephrase this as follows: do you expect the world that we are living in, and in particular the way that it responds to the actions that we take, to fundamentally be concave or convex? someone with a concave disposition might say things like this: "going to the extremes has never been good for us; you can die from being too hot or too cold. we need to find the balance between the two that's just right" "if you implement only a little bit of a philosophy, you can pick the parts that have the highest benefits and the lowest risks, and avoid the parts that are more risky. but if you insist on going to the extremes, once you've picked the low-hanging fruit, you'll be forced to look harder and harder for smaller and smaller benefits, and before you know it the growing risks might outweigh the benefit of the whole thing" "the opposing philosophy probably has some value too, so we should try to combine the best parts of both, and definitely avoid doing things that the opposing philosophy considers to be extremely terrible, just in case" someone with a convex disposition might say things like this: "we need to focus. otherwise, we risk becoming a jack of all trades, master of none" "if we take even a few steps down that road, it will become slippery slope and only pull us down ever further until we end up in the abyss. there's only two stable positions on the slope: either we're down there, or we stay up here" "if you give an inch, they will take a mile" "whether we're following this philosophy or that philosophy, we should be following some philosophy and just stick to it. making a wishy-washy mix of everything doesn't make sense" i personally find myself perenially more sympathetic to the concave approach than the convex approach, across a wide variety of contexts. if i had to choose either (i) a coin-flip between anarcho-capitalism and soviet communism or (ii) a 50/50 compromise between the two, i would pick the latter in a heartbeat. i argued for moderation in bitcoin block size debates, arguing against both 1-2 mb small blocks and 128 mb "very big blocks". i've argued against the idea that freedom and decentralization are "you either have it or you don't" properties with no middle ground. i argued in favor of the dao fork, but to many people's surprise i've argued since then against similar "state-intervention" hard forks that were proposed more recently. as i said in 2019, "support for szabo's law [blockchain immutability] is a spectrum, not a binary". but as you can probably tell by the fact that i needed to make those statements at all, not everyone seems to share the same broad intuition. i would particularly argue that the ethereum ecosystem in general has a fundamentally concave temperament, while the bitcoin ecosystem's temperament is much more fundamentally convex. in bitcoin land, you can frequently hear arguments that, for example, either you have self-sovereignty or you don't, or that any system must have either a fundamentally centralizing or a fundamentally decentralizing tendency, with no possibility halfway in between. the occasional half-joking support for tron is a key example: from my own concave point of view, if you value decentralization and immutability, you should recognize that while the ethereum ecosystem does sometimes violate purist conceptions of these values, tron violates them far more egregiously and without remorse, and so ethereum is still by far the more palatable of the two options. but from a convex point of view, the extremeness of tron's violations of these norms is a virtue: while ethereum half-heartedly pretends to be decentralized, tron is centralized but at least it's proud and honest about it. this difference between concave and convex mindsets is not at all limited to arcane points about efficiency/decentralization tradeoffs in cryptocurrencies. it applies to politics (guess which side has more outright anarcho-capitalists), other choices in technology, and even what food you eat. but in all of these questions too, i personally find myself fairly consistently coming out on the side of balance. being concave about concavity but it's worth noting that even on the meta-level, concave temperament is something that one must take great care to avoid being extreme about. there are certainly situations where policy a gives a good result, policy b gives a worse but still tolerable result, but a half-hearted mix between the two is worst of all. the coronavirus is perhaps an excellent example: a 100% effective travel ban is far more than twice as useful as a 50% effective travel ban. an effective lockdown that pushes the r0 of the virus down below 1 can eradicate the virus, leading to a quick recovery, but a half-hearted lockdown that only pushes the r0 down to 1.3 leads to months of agony with little to show for it. this is one possible explanation for why many western countries responded poorly to it: political systems designed for compromise risk falling into middle approaches even when they are not effective. another example is a war: if you invade country a, you conquer country a, if you invade country b, you conquer country b, but if you invade both at the same time sending half your soldiers to each one, the power of the two combined will crush you. in general, problems where the effect of a response is convex are often places where you can find benefits of some degree of centralization. but there are also many places where a mix is clearly better than either extreme. a common example is the question of setting tax rates. in economics there is the general principle that deadweight loss is quadratic: that is, the harms from the inefficiency of a tax are proportional to the square of the tax rate. the reason why this is the case can be seen as follows. a tax rate of 2% deters very few transactions, and even the transactions it deters are not very valuable how valuable can a transaction be if a mere 2% tax is enough to discourage the participants from making it? a tax rate of 20% would deter perhaps ten times more transactions, but each individual transaction that was deterred is itself ten times more valuable to its participants than in the 2% case. hence, a 10x higher tax may cause 100x higher economic harm. and for this reason, a low tax is generally better than a coin flip between high tax and no tax. by similar economic logic, an outright prohibition on some behavior may cause more than twice as much harm as a tax set high enough to only deter half of people from participating. replacing existing prohibitions with medium-high punitive taxes (a very concave-temperamental thing to do) could increase efficiency, increase freedom and provide valuable revenue to build public goods or help the impoverished. another example of effects like this in laffer curve: a tax rate of zero raises no revenue, a tax rate of 100% raises no revenue because no one bothers to work, but some tax rate in the middle raises the most revenue. there are debates about what that revenue-maximizing rate is, but in general there's broad agreement that the chart looks something like this: if you had to pick either the average of two proposed tax plans, or a coin-flip between them, it's obvious that the average is usually best. and taxes are not the only phenomenon that are like this; economics studies a wide array of "diminishing returns" phenomena which occur everywhere in production, consumption and many other aspects of regular day-to-day behavior. finally, a common flip-side of diminishing returns is accelerating costs: to give one notable example, if you take standard economic models of utility of money, they directly imply that double the economic inequality can cause four times the harm. the world has more than one dimension another point of complexity is that in the real world, policies are not just single-dimensional numbers. there are many ways to average between two different policies, or two different philosophies. one easy example to see this is: suppose that you and your friend want to live together, but you want to live in toronto and your friend wants to live in new york. how would you compromise between these two options? well, you could take the geographic compromise, and enjoy your peaceful existence at the arithmetic midpoint between the two lovely cities at.... this assembly of god church about 29km southwest of ithaca, ny. or you could be even more mathematically pure, and take the straight-line midpoint between toronto and new york without even bothering to stay on the earth's surface. then, you're still pretty close to that church, but you're six kilometers under it. a different way to compromise is spending six months every year in toronto and six months in new york and this may well be an actually reasonable path for some people to take. the point is, when the options being presented to you are more complicated than simple single-dimensional numbers, figuring out how to compromise between the options well, and really take the best parts of both and not the worst parts of both, is an art, and a challenging one. and this is to be expected: "convex" and "concave" are terms best suited to mathematical functions where the input and the output are both one-dimensional. the real world is high-dimensional and as machine-learning researchers have now well established, in high-dimensional environments the most common setting that you can expect to find yourself in is not a universally convex or universally concave one, but rather a saddle point: a point where the local region is convex in some directions but concave in other directions. a saddle point. convex left-to-right, concave forward-to-backward. this is probably the best mathematical explanation for why both of these dispositions are to some extent necessary: the world is not entirely convex, but it is not entirely concave either. but the existence of some concave path between any two distant positions a and b is very likely, and if you can find that path then you can often find a synthesis between the two positions that is better than both. ethereum launches | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum launches posted by stephan tual on july 30, 2015 protocol announcements a few minutes ago, many of you generated and loaded the ethereum genesis block, marking the inception of frontier, the first live release of the ethereum project. the dev team joins me in thanking each and everyone of you for your support during the past year and half. it’s incredible to see the progress that has taken place since that humble first commit on christmas eve 2013. the vision of a censorship-proof ‘world computer’ that anyone can program, paying exclusively for what they use and nothing more, is now a reality. many of you will want to get started immediately. please check out our brand new website for instructions on how to install and run the clients. if you face issues, the first port of call should be our forums. for more detailed information as to what this release includes, please see my previous blog post. to celebrate the release, we’ll leave you with this new intro video to our project. we hope you enjoy it, and we cannot wait to see what you will build on ethereum! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements devcon one postponed until further notice | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search devcon one postponed until further notice posted by george hallam on september 2, 2015 events unfortunately we were not able to secure the venue for devcon one in london on the dates desired (october 5th-8th). as a consequence we are postponing the event until further notice.  please do not make travel plans at this time. updates will be forthcoming on the devcon one website . please standby for further information. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements devcon is back! | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search devcon is back! posted by vitalik buterin on september 24, 2015 events devcon 1 will be happening in london on november 9-13, a little over one hundred days since the ethereum network launched. over the last months, we’ve seen the network grow from a few hundred nodes starting on that one exciting and special night to a very substantial, globally deployed stable platform with thousands of devs pushing towards the decentralization revolution which motivates and inspires us. devcon will have three primary categories of topics: basic research and core protocols: including proof of stake, scalability, networking protocols, privacy and zero-knowledge proofs, as well as some of the more mathematical aspects of decentralized protocols like stablecoins, prediction markets and reputation systems. this part will be designed to be similar to an academic conference, in some ways in the spirit of cryptoeconomicon. dapp development: focusing on the practical challenges of developing applications on top of the ethereum platform and effective design patterns to optimize security, efficiency, developer time and the user experience. this part will be designed to be similar to a programming language developer convention, eg. http://nodejsconf.it/ industry and social implications: including iot, finance, government, supply chain tracking, notarization, identity and reputation. this part will be designed to appeal to industry and mainstream audiences, and particularly those from outside the ethereum ecosystem coming in with the goal of understanding what its features are and how their projects can benefit from using the technology, as well as other interested parties such as policymakers and investors. the conference will take place over 5 days from november 9-13 in london at the gibson hall. the first half of the week will focus on research and dapp development, and the second half of the week will focus on industry and social implications. we’ll also be discussing how development will continue now that the platform has launched, as well as the future direction of the foundation as an organization. we hope that over the course of the week you will be able to develop the understanding and make the connections you need to make the most of the opportunities ethereum presents. there will be keynotes from several of the core development team, updating you on progress and some exciting new projects, and as far as possible we’ll endeavor to livestream the event so that people who could not make it can come in too. one of the features of the week will be a focus on the dapps the community are producing a showcase, if you like, of all the great things happening on the ethereum platform. expect talks from development teams sharing what they have learned, and detailed talks about the new businesses and business models enabled by the ethereum platform. if you think that you have something to share, feel free to get in touch with us at info@ethereum.org. a website will soon be live at http://devcon.ethereum.org and we will add details on prices, agenda, sponsorships, etc as info becomes available. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle la guía incompleta de los rollups 2021 jan 05 see all posts gracias a j4c0b0bl4nd0n por la traducción los rollups están de moda en la comunidad de ethereum y están destinados a ser la solución de escalabilidad clave para ethereum en el futuro previsible. pero, ¿qué es exactamente esta tecnología, qué podemos esperar de ella y cómo podremos usarla? este post intentará responder algunas de esas preguntas clave. antecedentes: ¿qué es el escalado de capa-1 y capa-2? hay dos formas de escalar un ecosistema blockchain. primero, puede hacer que la propia cadena de bloques tenga una mayor capacidad de transacción. el principal desafío con esta técnica es que las cadenas de bloques con "bloques más grandes" son inherentemente más difíciles de verificar y es probable que se vuelvan más centralizadas. para evitar tales riesgos, los desarrolladores pueden aumentar la eficiencia del software del cliente o, de manera más sostenible, utilizar técnicas como la fragmentación para permitir que el trabajo de construcción y verificación de la cadena se divida en muchos nodos; el esfuerzo conocido como "eth2" actualmente está construyendo esta actualización en ethereum. en segundo lugar, puede cambiar la forma en que usa la cadena de bloques. en lugar de poner toda la actividad en la cadena de bloques directamente, los usuarios realizan la mayor parte de su actividad fuera de la cadena en un protocolo de "capa-2". hay un contrato inteligente en la cadena, que solo tiene dos tareas: procesar depósitos y retiros, y verificar las pruebas de que todo lo que sucede fuera de la cadena sigue las reglas. hay varias formas de hacer estas pruebas, pero todas comparten la propiedad que verificar las pruebas en la cadena es mucho más económico que hacer el cálculo original fuera de la cadena. state channels vs plasma vs rollups los tres tipos principales de escalado de capa-2 son state channels, plasma y rollups. estos son tres paradigmas diferentes, con diferentes fortalezas y debilidades, y en este punto estamos bastante seguros de que toda la escala de capa-2 cae aproximadamente en estas tres categorías (aunque existen controversias de nombres en los bordes, por ejemplo, ver "validium"). cómo funcionan state channels? ver también: https://www.jeffcoleman.ca/state-channels y statechannels.org imaginemos que alice le ofrece una conexión a internet a bob, a cambio de que bob le pague $0,001 por megabyte. en lugar de realizar una transacción para cada pago, alice y bob utilizan el siguiente esquema de capa-2. primero, bob pone $ 1 (o algún eth o equivalente de moneda estable) en un contrato inteligente. para realizar su primer pago a alice, bob firma un "boleto" (un mensaje fuera de la cadena), que simplemente dice "$0.001", y se lo envía a alice. para hacer su segundo pago, bob firmaría otro boleto que dice "$0.002" y se lo enviaría a alice. y así sucesivamente para tantos pagos como sea necesario. cuando alice y bob terminen de realizar transacciones, alice puede publicar el boleto de mayor valor para encadenarlo, envuelto en otra firma suya. el contrato inteligente verifica las firmas de alice y bob, le paga a alice el monto del boleto de bob y le devuelve el resto a bob. si alice no está dispuesta a cerrar el canal (debido a malicia o falla técnica), bob puede iniciar un período de retiro (por ejemplo, 7 días); si alice no proporciona un boleto dentro de ese tiempo, entonces bob recupera todo su dinero. esta técnica es poderosa: se puede ajustar para manejar pagos bidireccionales, relaciones de contratos inteligentes (por ejemplo, alice y bob hacen un contrato financiero dentro del canal) y composición (si alice y bob tienen un canal abierto y también bob y charlie, alice puede interactuar sin confianza con charlie). pero hay límites a lo que pueden hacer los canales. los canales no se pueden usar para enviar fondos fuera de la cadena a personas que aún no son participantes. los canales no se pueden utilizar para representar objetos que no tengan un propietario lógico claro (p. ej., uniswap). y los canales, especialmente si se utilizan para hacer cosas más complejas que simples pagos recurrentes, requieren una gran cantidad de capital para bloquearse. cómo funciona plasma? ver también: el paper original de plasma, y plasma cash. para depositar un activo, un usuario lo envía al contrato inteligente que administra la cadena plasma. la cadena plasma asigna a ese activo una nueva identificación única (por ejemplo, 537). cada cadena de plasma tiene un operador (podría ser un actor centralizado, multisig o algo más complejo como pos o dpos). cada intervalo (esto podría ser de 15 segundos, una hora o cualquier intervalo), el operador genera un "batch" que consta de todas las transacciones de plasma que han recibido fuera de la cadena. generan un árbol de merkle, en el que en cada índice "x" del árbol hay una transacción que transfiere el id de activo "x" si existe tal transacción y, de lo contrario, esa hoja es cero. publican el merkle root de este árbol a la cadena. también envían la rama merkle de cada índice x al propietario actual de ese activo. para retirar un activo, un usuario publica la sucursal de merkle de la transacción más reciente que le envía el activo. el contrato inicia un período de impugnación, durante el cual cualquiera puede intentar usar otras sucursales de merkle para invalidar la salida al demostrar que (i) el remitente no era propietario del activo en el momento en que lo envió, o (ii) envió el activo a otra persona en algún momento posterior. si nadie prueba que la salida es fraudulenta durante (por ejemplo) 7 días, el usuario puede retirar el activo. plasma proporciona propiedades más sólidas que los state channels: puede enviar activos a participantes que nunca formaron parte del sistema y los requisitos de capital son mucho más bajos. pero tiene un costo: los canales no requieren datos en absoluto para conectarse en cadena durante el "funcionamiento normal", pero plasma requiere que cada cadena publique un hash a intervalos regulares. además, las transferencias de plasma no son instantáneas: hay que esperar a que finalice el intervalo y se publique el bloque. además, plasma y state channels comparten una debilidad clave en común: la teoría del juego detrás de por qué son seguros se basa en la idea de que cada objeto controlado por ambos sistemas tiene algún "propietario" lógico. si a ese propietario no le importa su activo, puede resultar en un resultado "no válido" que involucre a ese activo. esto está bien para muchas aplicaciones, pero es un factor decisivo para muchas otras (por ejemplo: uniswap). incluso los sistemas en los que se puede cambiar el estado de un objeto sin el consentimiento del propietario (por ejemplo: los sistemas basados en cuentas, en los que puede aumentar el saldo de alguien sin su consentimiento) no funcionan bien con plasma. todo esto significa que se requiere una gran cantidad de "razonamiento específico de la aplicación" en cualquier despliegue realista de plasma o state channels, y no es posible hacer un sistema plasma o state channels que solo simule el entorno completo de ethereum (o "el evm") . para solucionar este problema, llegamos a... rollups. rollups ver también: ethhub en optimistic rollups y zk rollups. plasma y state channels son esquemas de capa-2 "completos", en el sentido de que intentan mover tanto los datos como la computación fuera de la cadena. sin embargo, los problemas fundamentales de la teoría de juegos en torno a la disponibilidad de datos significan que es imposible hacer esto de manera segura para todas las aplicaciones. plasma y state channels evitan esto al basarse en una noción explícita de propietarios, pero esto les impide ser completamente generales. los rollups, por otro lado, son un esquema de capa-2 "híbrido". los rollups mueven el cómputo (y el almacenamiento de estado) fuera de la cadena, pero mantienen algunos datos por transacción adentro de la cadena. para mejorar la eficiencia, utilizan una gran cantidad de trucos de compresión sofisticados para reemplazar los datos con computación siempre que sea posible. el resultado es un sistema en el que la escalabilidad sigue estando limitada por el ancho de banda de datos de la cadena de bloques subyacente, pero en una proporción muy favorable: mientras que una transferencia de token erc20 de capa base de ethereum cuesta ~45000 gas, una transferencia de token erc20 en un rollup ocupa 16 bytes de espacio en cadena y costos por debajo de 300 de gas. el hecho de que los datos estén en la cadena es clave (nota: poner los datos "en ipfs" no funciona, porque ipfs no brinda consenso sobre si un dato determinado está disponible o no; los datos deben ir en una cadena de bloques). poner datos en la cadena y tener consenso sobre ese hecho permite que cualquier persona procese localmente todas las operaciones en el rollup si así lo desea, lo que les permite detectar fraudes, iniciar retiros o comenzar personalmente a producir batch de transacciones. la falta de problemas de disponibilidad de datos significa que un operador malintencionado o fuera de línea puede causar incluso menos daño (por ejemplo: no puede causar un retraso de 1 semana), lo que abre un espacio de diseño mucho más grande para quién tiene derecho a publicar batch y hace que los rollups sean mucho más sencillos. y lo que es más importante, la falta de problemas de disponibilidad de datos significa que ya no es necesario asignar activos a los propietarios, lo que lleva a la razón clave por la cual la comunidad de ethereum está mucho más entusiasmada con los rollups que con las formas anteriores de escalado de capa-2: los rollups son totalmente de multi-propósito, e incluso se puede ejecutar un evm dentro de un rollup, lo que permite que las aplicaciones ethereum existentes migren a rollups casi sin necesidad de escribir ningún código nuevo. ok, entonces, ¿cómo funciona exactamente un rollup? hay un contrato inteligente dentro de la cadena que mantiene una estado root: el merkle root del estado del rollup (es decir, los saldos de cuenta, el código del contrato, etc., lo que está "dentro" del rollup). cualquiera puede publicar un batch, una colección de transacciones en un formato altamente comprimido junto con el root del estado anterior y el root del nuevo estado (el merkle root después de procesar las transacciones). el contrato verifica que el root de estado anterior en el batch coincida con su root de estado actual; si lo hace, cambia el root del estado a el nuevo root del estado. para admitir depósitos y retiros, agregamos la capacidad de tener transacciones cuya entrada o salida está "fuera" del estado del rollup. si un batch tiene entradas desde el exterior, la transacción que envía el batch también debe transferir estos activos al contrato de rollup. si un batch tiene salidas al exterior, luego de procesar el batch, el contrato inteligente inicia esos retiros. ¡y eso es! excepto por un detalle importante: ¿cómo saber si los roots posteriores al estado en el batch son correctas? si alguien puede enviar un batch con cualquier root posterior al estado sin consecuencias, podría simplemente transferir todas las monedas que contiene el rollup a sí mismos. esta pregunta es clave porque hay dos familias de soluciones muy diferentes para el problema, y estas dos familias de soluciones conducen a los dos tipos de rollups. optimistic rollups vs zk rollups los dos tipos de rollups son: optimistic rollups, que utilizan pruebas de fraude: el contrato del rollup realiza un seguimiento de todo su historial de state roots y el hash de cada batch. si alguien descubre que un batch tenía un root posterior al estado incorrecta, puede publicar una prueba en cadena, demostrando que el batch se calculó incorrectamente. el contrato verifica la prueba y revierte ese batch y todos los batches posteriores. zk rollups, que usan pruebas de validez: cada batch incluye una prueba criptográfica llamada zk-snark (por ejemplo: usando el protocolo plonk), lo que demuestra que el root posterior al estado es el resultado correcto para ejecutar el batch. no importa cuán grande sea el cálculo, la prueba se puede verificar muy rápidamente en la cadena. hay compensaciones complejas entre los dos tipos de rollups: propiedad optimistic rollups zk rollups costo fijo de gas por batch ~40,000 (una transacción liviana que principalmente solo cambia el valor del root del estado) ~500,000 (la verificación de un zk-snark es bastante intensiva computacionalmente) tiempo de retiro ~1 semana (los retiros deben retrasarse para dar tiempo a que alguien publique una prueba de fraude y cancele el retiro si es fraudulento) muy rápida (solo espera el siguiente batch) complejidad de la tecnología baja alta (zk-snark es una tecnología muy nueva y matemáticamente compleja) generalizabilidad más fácil (los rollups de evm de propósito general ya están cerca de la red principal) más difícil (para zk-snark probar la ejecución de evm de propósito general es mucho más difícil que probar cálculos simples, aunque hay esfuerzos (por ejemplo: cairo) trabajando para mejorar esto) costos de gas en cadena por transacción altos bajos (si los datos en una transacción solo se usan para verificar, y no para causar cambios de estado, entonces estos datos pueden omitirse, mientras que en un optimist rollup tendrían que publicarse en caso de que deban verificarse en una prueba de fraude) costos de computación fuera de la cadena bajos (aunque hay más necesidad de muchos nodos completos para rehacer el cálculo) altos (la prueba de zk-snark, especialmente para el cómputo de propósito general, puede ser costosa, potencialmente miles de veces más costosa que ejecutar el cómputo directamente.) en general, mi opinión es que, a corto plazo, es probable que los optimist rollups ganen para el cómputo de evm de propósito general y que los zk rollups probablemente ganen para pagos simples, intercambios y otros casos de uso específicos de la aplicación, pero en los rollups de zk a mediano y largo plazo ganarán en todos los casos de uso a medida que mejore la tecnología zk-snark. anatomía de prueba de fraude la seguridad de un rollup de optimistic depende de la idea de que si alguien publica un batch inválido en el rollup, cualquier otra persona que se mantuvo al día con la cadena y detectó el fraude puede publicar una prueba de fraude, demostrando al contrato que ese batch no es válido y debe revertirse. una prueba de fraude que afirme que un batch no es válido contendría los datos en verde: el batch en sí (que podría verificarse con un hash almacenado en cadena) y las partes del árbol merkle necesarias para probar solo las cuentas específicas que se leyeron y/o modificado por el batch. los nodos del árbol en amarillo se pueden reconstruir a partir de los nodos en verde, por lo que no es necesario proporcionarlos. estos datos son suficientes para ejecutar el batch y calcular el root posterior al estado (tenga en cuenta que esto es exactamente igual a cómo los clientes sin estado verifican bloques individuales). si el root posterior al estado calculado y el root posterior al estado proporcionado en el batch no son iguales, entonces el batch es fraudulento. se garantiza que si un batch se construyó incorrectamente, y todos los batches anteriores se construyeron correctamente, entonces es posible crear una prueba de fraude que muestre que el batch se construyó incorrectamente. tenga en cuenta la afirmación sobre los batches anteriores: si hubo más de un batch no válido publicado en el rollup, entonces es mejor intentar probar que el primero no es válido. por supuesto, también se garantiza que si un batch se construyó correctamente, nunca será posible crear una prueba de fraude que demuestre que el batch no es válido. ¿cómo funciona la compresión? una transacción simple de ethereum (para enviar eth) requiere ~110 bytes. sin embargo, una transferencia eth en un rollup solo requiere ~12 bytes: parametro ethereum rollup nonce ~3 0 gasprice ~8 0-0.5 gas 3 0-0.5 to 21 4 value ~9 ~3 signature ~68 (2 + 33 + 33) ~0.5 from 0 (recovered from sig) 4 total ~112 ~12 parte de esto es simplemente una codificación superior: el rlp de ethereum desperdicia 1 byte por valor en la longitud de cada valor. pero también hay algunos trucos de compresión muy inteligentes que están sucediendo: nonce: el propósito de este parámetro es evitar repeticiones. si el nonce actual de una cuenta es 5, la próxima transacción de esa cuenta debe tener un nonce 5, pero una vez que se procese la transacción, el nonce en la cuenta se incrementará a 6 para que la transacción no se pueda procesar nuevamente. en el rollup, podemos omitir el nonce por completo, porque solo recuperamos el nonce del estado anterior; si alguien intenta reproducir una transacción con un nonce anterior, la firma no se verificará, ya que la firma se verificará con los datos que contienen el nuevo nonce superior. gasprice: podemos permitir que los usuarios paguen con un rango fijo de precios de gas, por ejemplo: una elección de 16 potencias consecutivas de dos. alternativamente, podríamos simplemente tener un nivel de tarifa fijo en cada batch, o incluso mover el pago de gas fuera del protocolo de acumulación por completo y hacer que los transactores paguen a los creadores de batch para su inclusión a través de un canal. gas: de manera similar, podríamos restringir el gas total a una elección de potencias consecutivas de dos. alternativamente, podríamos tener un límite de gas solo a nivel de batch. to: podemos reemplazar la dirección de 20 bytes con un index (por ejemplo, si una dirección es la dirección 4527 agregada al árbol, solo usamos el index 4527 para referirnos a ella. agregaríamos un subárbol al estado para almacenar el mapeo de índices a direcciones). value: podemos almacenar valor en notación científica. en la mayoría de los casos, las transferencias solo necesitan de 1 a 3 dígitos significativos. signature: podemos usar bls firmas agregadas, que nos permite agregar muchas firmas en una sola firma de ~32-96 bytes (según el protocolo). luego, esta firma se puede verificar con el conjunto completo de mensajes y remitentes en un batch, todo a la vez. el ~0.5 en la tabla representa el hecho de que existe un límite en la cantidad de firmas que se pueden combinar en un agregado que se puede verificar en un solo bloque, por lo que los batches grandes necesitarían una firma por cada ~100 transacciones. un truco de compresión importante que es específico de los rollups de zk es que si una parte de una transacción solo se usa para la verificación y no es relevante para calcular la actualización del estado, entonces esa parte se puede dejar fuera de la cadena. esto no se puede hacer en un rollup de optimistic porque esos datos aún deberían incluirse en la cadena en caso de que deban verificarse más tarde en una prueba de fraude, mientras que en un rollup de zk, el snark que prueba la corrección del batch ya prueba que cualquier dato necesario para la verificación. un ejemplo importante de esto son los rollup que preservan la privacidad: en un rollup de optimistic, el zk-snark de ~500 bytes utilizado para la privacidad en cada transacción debe ir en cadena, mientras que en un rollup de zk, el zk-snark que cubre todo el batch ya no deja duda que los zk-snark "internos" sean válidos. estos trucos de compresión son clave para la escalabilidad de los rollups; sin ellos, los rollups serían quizás solo una mejora de ~10x en la escalabilidad de la cadena base (aunque hay algunas aplicaciones específicas de computación pesada donde incluso los resúmenes simples son poderosos), mientras que con los trucos de compresión, el factor de escala puede superar 100x por casi todas las aplicaciones. ¿quién puede enviar un batch? hay varias escuelas de pensamiento sobre quién puede enviar un batch en un rollup optimistic o zk. en general, todos están de acuerdo en que para poder enviar un batch, un usuario debe hacer un depósito grande; si ese usuario alguna vez envía un batch fraudulento (por ejemplo, con un root de estado no válido), ese depósito se quemará en parte y se entregará en parte como recompensa al probador de fraude. pero más allá de eso, hay muchas posibilidades: anarquía total: cualquiera puede enviar un batch en cualquier momento. este es el enfoque más simple, pero tiene algunos inconvenientes importantes. en particular, existe el riesgo de que varios participantes generen e intenten enviar batches en paralelo, y solo uno de esos batches se puede incluir con éxito. esto conduce a una gran cantidad de esfuerzo desperdiciado en la generación de pruebas y/o gas desperdiciado en la publicación de batch en cadena. secuenciador centralizado: hay un solo actor, el secuenciador, que puede enviar batches (con la excepción de los retiros: la técnica habitual es que un usuario primero puede enviar una solicitud de retiro y luego, si el secuenciador no procesa ese retiro en el siguiente batch, entonces el usuario puede enviar un batch de una sola operación por sí mismo). este es el más "eficiente", pero depende de un actor central para la vitalidad. subasta de secuenciador: se lleva a cabo una subasta (por ejemplo: todos los días) para determinar quién tiene derecho a ser el secuenciador para el día siguiente. esta técnica tiene la ventaja de que recauda fondos que podrían ser distribuidos por ejemplo: un dao controlado por el rollup (ver: subastas mev) selección aleatoria desde el conjunto pos: cualquiera puede depositar eth (o quizás el token de protocolo propio del rollup) en el contrato del rollup, y el secuenciador de cada batch se selecciona aleatoriamente de uno de los depositantes, con la probabilidad de ser seleccionado proporcional a la cantidad depositada. el principal inconveniente de esta técnica es que conduce al bloqueo de grandes cantidades de capital innecesarias. votación dpos: solo hay un secuenciador seleccionado con una subasta, pero si su rendimiento es bajo, los poseedores de tokens pueden votar para expulsarlos y realizar una nueva subasta para reemplazarlos. procesamiento por batches dividido y aprovisionamiento de state root algunos de los rollups que se están desarrollando actualmente usan un paradigma de "batch dividido", donde la acción de enviar un batch de transacciones de capa-2 y la acción de enviar una state root se realizan por separado. esto tiene algunas ventajas clave: puede permitir que muchos secuenciadores publiquen batches en paralelo para mejorar la resistencia a la censura, sin preocuparse de que algunos batches no sean válidos porque se incluyó otro primero. si un state root es fraudulento, no necesita revertir todo el batch; puede revertir solo el root del estado y esperar a que alguien proporcione un nuevo root del estado para el mismo batch. esto brinda a los remitentes de transacciones una mejor garantía de que sus transacciones no se revertirán. entonces, en general, hay un zoológico bastante complejo de técnicas que intentan equilibrar complicadas compensaciones que involucran eficiencia, simplicidad, resistencia a la censura y otros objetivos. todavía es demasiado pronto para decir qué combinación de estas ideas funciona mejor; el tiempo dirá. ¿cuánta escalabilidad te dan los rollups? en la cadena ethereum existente, el límite de gas es de 12,5 millones, y cada byte de datos en una transacción cuesta 16 de gas. esto significa que si un bloque no contiene nada más que un solo batch (diremos que se usa un rollup de zk, gastando 500k de gas en la verificación de prueba), ese batch puede tener (12 millones/16) = 750,000 bytes de datos. como se muestra arriba, un rollup para transferencias eth requiere solo 12 bytes por operación de usuario, lo que significa que el batch puede contener hasta 62 500 transacciones. con un tiempo de bloque promedio de 13 segundos, esto se traduce en ~4807 tps (en comparación con 12,5 millones/21000/13 ~= 45 tps para transferencias eth directamente en ethereum). aquí hay una tabla para algunos otros casos de uso de ejemplo: | applicación | bytes en rollup | costo gas en capa-1 | max aumento escalabilidad | | | | | | | eth transferencia | 12 | 21,000 | 105x | | erc20 transferencia | 16 (4 bytes más para especificar qué token) | ~50,000 | 187x | | uniswap trade | ~14 (remitente de 4 bytes + destinatario de 4 bytes + valor de 3 bytes + precio máximo de 1 byte + miscelánea de 1 byte) | ~100,000 | 428x | | retiro preservando la privacidad (rollup de optimistic) | 296 (4 bytes index root + 32 bytes anulador + 4 bytes destinatario + 256 bytes prueba zk-snark | ~380,000 | 77x la ganancia máxima de escalabilidad se calcula como (l1 costo gas) / (bytes enrrollup * 16) * 12 million / 12.5 millones. ahora, vale la pena tener en cuenta que estas cifras son demasiado optimistas por varias razones. lo que es más importante, un bloque casi nunca contendría solo un batch, al menos porque hay y habrá múltiples rollups. segundo, los depósitos y retiros seguirán existiendo. en tercer lugar, en el corto plazo el uso será bajo, por lo que dominarán los costos fijos. pero incluso teniendo en cuenta estos factores, se espera que las ganancias de escalabilidad de más de 100x sean la norma. ahora, ¿qué sucede si queremos superar ~1000-4000 tps (según el caso de uso específico)? aquí es donde entra en juego la fragmentación de datos eth2. la propuesta de fragmentación abre un espacio de 16 mb cada 12 segundos que se puede llenar con cualquier dato, y el sistema garantiza el consenso sobre la disponibilidad de esos datos. este espacio de datos puede ser utilizado por los rollups. estos ~1398k bytes por segundo son una mejora de 23 veces sobre los ~60 kb/seg de la cadena ethereum existente y, a largo plazo, se espera que la capacidad de datos crezca aún más. por lo tanto, los rollups que utilizan datos fragmentados de eth2 pueden procesar colectivamente hasta ~100 000 tps, e incluso más en el futuro. ¿cuáles son algunos de los desafíos que aún no se han resuelto por completo en los rollups? si bien el concepto básico de un rollup ahora se comprende bien, estamos bastante seguros de que son fundamentalmente factibles y seguros, y ya se han implementado varios rollups en la red principal, todavía hay muchas áreas del diseño del rollup que no se han explorado bien, y bastantes desafíos para llevar por completo grandes partes del ecosistema ethereum a rollups para aprovechar su escalabilidad. algunos desafíos clave incluyen: incorporación de usuarios y ecosistemas no muchas aplicaciones usan rollups, los rollups no son familiares para los usuarios y pocas billeteras han comenzado a integrar rollups. los comerciantes y las organizaciones benéficas aún no los aceptan para pagos. transacciones cruzadas mover activos y datos de manera eficiente (por ejemplo: salidas de oracúlo) de un rollup a otro sin incurrir en el gasto de pasar por la capa base. incentivos de auditoría ¿cómo maximizar la posibilidad de que al menos un nodo honesto realmente verifique completamente un rollup de optimistic para que puedan publicar una prueba de fraude si algo sale mal? para rollups a pequeña escala (hasta unos pocos cientos de tps), este no es un problema importante y uno simplemente puede confiar en el altruismo, pero para rollups a mayor escala se necesita un razonamiento más explícito sobre esto. explorando el espacio de diseño entre plasma y rollups ¿existen técnicas que pongan algunos datos relevantes para la actualización del estado en la cadena, pero no todos, y hay algo útil que pueda resultar de eso? maximizar la seguridad de las confirmaciones previas muchos rollups brindan una noción de "confirmación previa" para una ux más rápida, donde el secuenciador promete de inmediato que una transacción se incluirá en el siguiente batch, y el depósito del secuenciador se destruye si no cumplen su palabra. pero la seguridad económica de este esquema es limitada, debido a la posibilidad de hacer muchas promesas a muchos actores al mismo tiempo. ¿se puede mejorar este mecanismo? mejora de la velocidad de respuesta a los secuenciadores ausentes si el secuenciador de un rollup se desconecta repentinamente, sería valioso recuperarse de esa situación de la manera más rápida y económica, ya sea saliendo en masa de manera rápida y económica a un rollup diferente o reemplazando el secuenciador. eficiente zk-vm generando una prueba zk-snark de que el código evm de propósito general (o alguna vm diferente en la que se puedan compilar los contratos inteligentes existentes) se ha ejecutado correctamente y tiene un resultado determinado. conclusiones los rollups son un nuevo y poderoso paradigma de escalamiento de capa-2, y se espera que sean la piedra angular del escalamiento de ethereum en el futuro a corto y mediano plazo (y posiblemente también a largo plazo). han visto una gran cantidad de entusiasmo por parte de la comunidad ethereum porque, a diferencia de los intentos anteriores de escalado de capa-2, pueden admitir código evm de uso general, lo que permite que las aplicaciones existentes se migren fácilmente. lo hacen al hacer un compromiso clave: no intentar salir completamente de la cadena, sino dejar una pequeña cantidad de datos por transacción en la cadena. hay muchos tipos de resúmenes y muchas opciones en el espacio de diseño: uno puede tener un rollup de optimistic usando pruebas de fraude o un rollup de zk usando pruebas de validez (también conocido como zk-snark). el secuenciador (el usuario que puede publicar batch de transacciones para encadenar) puede ser un actor centralizado, un free-for-all o muchas otras opciones intermedias. los rollups aún son una tecnología en etapa inicial y el desarrollo continúa rápidamente, pero funcionan y algunos (en particular, loopring, zksync y deversifi) ya se han estado ejecutando durante meses. esperamos un trabajo mucho más emocionante que surja del espacio acumulativo en los años venideros. applications of security deposits and prediction markets you might not have thought about | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search applications of security deposits and prediction markets you might not have thought about posted by vitalik buterin on november 24, 2015 research & development special thanks to vlad zamfir for his work in developing many of the ideas behind prediction markets for content curation. for the past six years, people have been searching to try to find those elusive applications of blockchain technology that could finally break out into the mainstream. for cryptocurrency, the applications are largely already known though it of course remains to be seen just how well it will be able to retain its advantages as traditional payment systems continue to become more efficient. but what about smart contracts and crypto 2.0? one route that we can take is to simply look where legal contracts are being used today, and see where we can increase efficiencies by "smart-ifying" as many of them as possible. another route, however, is the peter thiel-ian zero-to-one approach: see if it's possible to use these tools to create industries that currently do not exist. hence, let us take a detour along that path, and see some of the more underappreciated and interesting applications that smart contracts can provide. hashcash meets proof of stake although proof of work is today known primarily for its function in blockchain consensus algorithms, its original function was in fact something quite different. the first major use of proof of work was adam back's hashcash, a protocol that tries to fight email spam by making spam emails more expensive to generate. proof of stake is today similarly best known for its applications in blockchain consensus, and the theory is that because of the way modern proof of stake works allowing users to put at stake large amounts of economic resources for security without, in the normal case, actually spending them it could potentially be vastly more efficient. so this leads to an interesting question: can we apply these advantages to make a more efficient version of hashcash using proof of stake as well? let us start by describing the problem. the basic principle behind hashcash rests on the idea that email today tends to be split into two categories: desirable email ("ham"), which people spend a reasonably large amount of effort writing and which people derive high value from reading, and undesirable email ("spam"), which people normally spend a much smaller amount of effort per email writing and which people derive a negative amount of value from reading. hence, the theory goes, by attaching a small mandatory cost to each email sent, the "ham" can get through with only slight additional burden, whereas "spam" would be made entirely unprofitable. this "cost" is paid in the form of electricity and computational effort which is used to solve a mathematical puzzle that can be quickly verified by the recipient's client before showing the email to the recipient. the problem with this approach is threefold. first, spammers may simply switch to a strategy of spending slightly more effort on each email (eg. spending five seconds of human labor per message to make it more likely to catch the reader's attention or bypass spam filters), and the threshold of computational work that would be needed to actually stop the majority of spam would be quite large. second, spammers are more professional and have better access to specialized computer hardware that can quickly and cheaply solve these computational puzzles, so a cost that might be five cents to the ordinary user might only be a tenth of a cent to a spammer. third, as this now-famous checklist listing flaws in popular proposed solutions to email spam points out, it is considered highly desirable to have a system where "sending email should be free". so here is a proof-of-stake alternative. when you send an email, you do not need to compute an expensive computational puzzle; instead, send a transaction to the blockchain which creates a contract containing some amount of money as a security deposit. as part of the email, send a private key to the recipient, which the recipient can submit into the contract to destroy the deposit (or donate it to a standardized charity) if they want to. if the deposit is not destroyed for some number of days, it is refunded to the sender. note that there would be no gain to the recipient in destroying the deposit the only motivation to do so would be pure spite. as a result, we get an asymmetry: the average cost for normal people to send an email would be small, because the recipient would only click "report spam" in the rare case that they are malicious, but the average cost for spammers would be quite high indeed and specialized hardware asymmetries would not help spammers one bit. one could see deposits going as high as a dollar in size, and one can even adopt a graduated scheme: senders can send whatever deposits they want above some minimum, but the level of notification that the recipient sees would depend on the exact amount. if it's 0.1,thenit′sjustanemailintheirmailbox.ifit′s0.1, then it's just an email in their mailbox. if it's 0.1,thenit′sjustanemailintheirmailbox.ifit′s1, then a few phone notifications. if it's $500, then their phone would ring at maximum volume overriding all other settings but the sender best be prepared to pay the price if the recipient deems the sender's intrusion unjustified. one can create more advanced versions of this scheme that do not require sending a transaction to create a new deposit for each email; one can imagine a scheme where the sender sends out many keys to destroy portions of the same security deposit, alongside signatures saying that those keys are valid, and recipients publish the signatures (but not the keys!) to a whisper-like channel allowing them to quickly sample and make sure that a particular deposit is not "over-subscribed" with destruction keys (one precise mechanism for doing this is to only treat signatures containing an index from 1 to n as valid, and add a rule that states that two signatures with the same index can be submitted to destroy the entire deposit with 10% transferred to the submitter; hence, one can be fairly sure that at most n signatures for that deposit exist). this would reduce the transaction load to roughly something like one transaction per email sender per year. in any case, smart contracts offer near-infinite room for creativity in optimizing the details. prediction markets and reddit one of the largest debates in online communities like reddit is the question of exactly how much centralized moderation is justified. one view is that the very power of the internet comes in large part from its egalitarian decentralized nature, and the fact that no single party has a higher class of authority than any other. some people may be more influential than others, but (i) that is a difference of degree and not a categorical distinction of class, and (ii) it is fundamentally the audience's choice to be influenced. the other view is that without centralized moderation, communities inevitably collapse into mediocrity and chaos of the undesirable kind; essentially, eternal september, and so having a small number of users ultimately in charge is, as is in many places the case, a "necessary evil". in practice, community voting moderation is quite powerful, but the centralist view also seems to have some merit. although comments that a community does not want to see do eventually get voted down, at least on reddit specifically the process takes time, and there is still a period of one or two hours during which such content remains on the front page. within a voting framework, to some extent this is unavoidable: if it was possible for a barrage of downvotes to very quickly remove content from the front page, that itself would turn into a censorship vector for vocal minorities. however, what if there was a third way to solve this problem, by using our favorite governance mechanism: prediction markets? prediction markets have so far, including by myself, often been introduced as a governance mechanism that could one day be employed for very large-scale decisions: whether or not we should bail out the banks, hire or fire a particular ceo, or enter a particular trade agreement. however, perhaps it might be better to introduce prediction markets into the world as a tool for decisions which are far more small-scale and non-threatening perhaps on the scale of hundreds or tens of dollars, or even ten cents. one could imagine a design that works as follows. rather than simply being votes, upvotes and downvotes to a comment on a hypothetical predictionreddit would be bets on a prediction market specific to that comment. the prediction market would be seeded by a mandatory bet that would need to be made by the person making the comment that their comment will be accepted as good; from there, upvotes and downvotes would shift the "price" of the market depending on how people vote. 99% of the time, the market would have no effect except that comments with high prices would be shown more prominently on the interface; the remaining 1% of the time, however, the comment would be submitted to a meta-moderation panel, which would vote on whether the comment is good or bad (or perhaps some score in between), and the participants in the prediction market would be compensated appropriately based on how well they predicted this score. the meta-moderation panel could in principle be quite large; potentially every single participant in the community could be included, provided an effective anti-sybil mechanism was in place. even a schellingcoin oracle could be used. it also does not have to be the case that 99% of markets are discarded; one could instead have a model where all markets are processed, but only a small portion of the meta-moderation panel sees each individual post; the number of people need only be large enough that they cannot practically collude for the purpose of insider trading the prediction markets. another alternative is to have the size or probability of meta-moderation be proportional to the volume of the market, so that the posts that receive the most attention are the ones where the stakes are highest. in any case, this particular means of combining reddit and cryptocurrency seems at least a bit more promising than simply integrating the ability to express condolences for someone's death by micro-tipping their relatives three cents. in principle, either of these two models could be extended quite a bit: imagine ads that are more expensive to maintain the more annoying they are to viewers, or a decentralized search engine where anyone can "plug in" their ranking algorithms by participating in the prediction markets, and profiting only if the algorithms are effective. oleg andreev's 2-of-2 escrow could be augmented with a reputation system via a prediction market on the probability that the escrow deposit will be destroyed or payment delayed. just remember, security deposits and prediction markets are essentially equivalent: a prediction market is a security deposit where anyone can challenge and require a higher deposit in response, and anyone else can back the original depositor up, and a security deposit is a prediction market where one particular party is forced to make a mandatory bet. perhaps this is a large part of the potential of what crypto 2.0 technology can offer: bring the internet from simple information technologies to economic information technologies that could potentially radically increase efficiencies, at least in a few sectors of the digital economy, by using incentives to more cleverly elicit the information that we all individually have. in any case, let's build these tools and find out. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum wallet developer preview | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum wallet developer preview posted by fabian vogelsteller on september 16, 2015 research & development we are happy to announce our very first developer-preview of the ethereum wallet ðapp. the point of this release is to gather feedback, squash bugs and, most importantly, get the code audited. please note that this is a developer-preview and not the final release. we advise you to be extremely careful putting large amount of ether in the wallet contracts. using the wallet on the mainnet should only be done with small amounts!   as steve ballmer once said developers! developers! developers! and note that this is exactly our target audience, don’t blindly trust us and we ask (and advise!) you to take a thorough look through the code in the ethereum wallet repository! if you’d like to build the wallet yourself, you need to head over to the mist repository, use the wallet branch and follow the instructions in the readme. reporting issues if you have any issues with the wallet, open the developer console of the wallet (menu -> developer -> toggle console) and provide the logs from there and the terminal where you started geth  or eth from. please report all issues to the wallet repository. how to run it? first download the binary for you os: (**note**: you can find newer releases here) mac os x windows 64bit windows 32bit (though we probably won't support bundled nodes with 32bit) linux 64bit linux 32bit this developer preview doesn't come bundled with a node, as there are a few things still to be finalised, so you still need to start one yourself. for this developer preview the supported clients are geth and eth. python is currently not supported because it does not have the required ipc interface to run the wallet. if you don't have one of these nodes installed yet, follow the instructions here or download a pre-build version. make sure you have updated to the latest version.  and start a node by simply running: go: $ geth if you want to unlock an account to be able to transfer add --unlock <my account>, or start a console with $ geth attach and unlock it  using the javascript interface: personal.unlockaccount('<my account>'). c++: $ eth it is important to note that the wallet is expecting a fully synced up node. in future versions of geth and eth the wallet will make use of the new eth_syncing method in the json rpc, allowing you to see a sync screen when you start the wallet. this feature is currently already supported by geth and eth on their develop branches. finally start the wallet by clicking the executable! running on a testnet if you want to try the wallet on a testnet you need to start your node with a different network id and probably a different data directory. to make sure the wallet can still connect to your node you manually need to set the ipc path: os x: $ geth --networkdid "1234" --datadir "/some/other/path" --ipcpath "/users/<username>/library/ethereum/geth.ipc" linux: $ geth --networkdid "1234" --datadir "/some/other/path" --ipcpath "/home/<username>/.ethereum/geth.ipc" additional you should probably provide your own genesis block using the --genesis flag. for more details about the flags see the wiki. after the node is started you can simple start the wallet again. note that you need to wait sometimes a bit, and click in the button in the corner. once you opened the wallet you will see a popup asking you to deploy a wallet contract on your testnet, which will be used as a code basis for your future wallet contracts. the main advantage is that it is much cheaper (1.8mio vs 180k gas). note: make sure you have the displayed account unlocked and has at least 1 ether. using the wallet the wallet allows you to create two types of  wallets: a simple wallet - works like a normal account (additional features are being worked on; e.g. adding owners, setting a daily limit) a multisig wallet - allows you to add any number of owner accounts and set a daily limit. every owner can send money from that account as long as it is under the daily limit. if above you need the signatures of the required other owners. when operating on the main net make sure you write down / backup the wallet contract address! this address is required in case you need to reimport your wallet on a different computer or during backup/recovery. multisig if you want to send and amount which is over the daily limit, your other owners need to sign. this should mostly be done from another computer, though you could as well add accounts you have in the same node. if a pending request comes in it will look as follows: simply click approve and the transaction goes through. deleting wallets if you’d like to delete a wallet click the trash icon on the wallet page, next to the wallet name. after you typed the name of the wallet it will be deleted from the ðapp. if you wrote the address down, you can always re-import the wallet in the "add wallet" section. roadmap when everything works fine and we finished the binary integration we are planning to release a first official version in 1-2 weeks™ until then please file issues and discuss it on reddit! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle deeper dive on cross-l2 reading for wallets and other use cases 2023 jun 20 see all posts special thanks to yoav weiss, dan finlay, martin koppelmann, and the arbitrum, optimism, polygon, scroll and soulwallet teams for feedback and review. in this post on the three transitions, i outlined some key reasons why it's valuable to start thinking explicitly about l1 + cross-l2 support, wallet security, and privacy as necessary basic features of the ecosystem stack, rather than building each of these things as addons that can be designed separately by individual wallets. this post will focus more directly on the technical aspects of one specific sub-problem: how to make it easier to read l1 from l2, l2 from l1, or an l2 from another l2. solving this problem is crucial for implementing an asset / keystore separation architecture, but it also has valuable use cases in other areas, most notably optimizing reliable cross-l2 calls, including use cases like moving assets between l1 and l2s. recommended pre-reads post on the three transitions ideas from the safe team on holding assets across multiple chains why we need wide adoption of social recovery wallets zk-snarks, and some privacy applications dankrad on kzg commitments verkle trees table of contents what is the goal? what does a cross-chain proof look like? what kinds of proof schemes can we use? merkle proofs zk snarks special purpose kzg proofs verkle tree proofs aggregation direct state reading how does l2 learn the recent ethereum state root? wallets on chains that are not l2s preserving privacy summary what is the goal? once l2s become more mainstream, users will have assets across multiple l2s, and possibly l1 as well. once smart contract wallets (multisig, social recovery or otherwise) become mainstream, the keys needed to access some account are going to change over time, and old keys would need to no longer be valid. once both of these things happen, a user will need to have a way to change the keys that have authority to access many accounts which live in many different places, without making an extremely high number of transactions. particularly, we need a way to handle counterfactual addresses: addresses that have not yet been "registered" in any way on-chain, but which nevertheless need to receive and securely hold funds. we all depend on counterfactual addresses: when you use ethereum for the first time, you are able to generate an eth address that someone can use to pay you, without "registering" the address on-chain (which would require paying txfees, and hence already holding some eth). with eoas, all addresses start off as counterfactual addresses. with smart contract wallets, counterfactual addresses are still possible, largely thanks to create2, which allows you to have an eth address that can only be filled by a smart contract that has code matching a particular hash. eip-1014 (create2) address calculation algorithm. however, smart contract wallets introduce a new challenge: the possibility of access keys changing. the address, which is a hash of the initcode, can only contain the wallet's initial verification key. the current verification key would be stored in the wallet's storage, but that storage record does not magically propagate to other l2s. if a user has many addresses on many l2s, including addresses that (because they are counterfactual) the l2 that they are on does not know about, then it seems like there is only one way to allow users to change their keys: asset / keystore separation architecture. each user has (i) a "keystore contract" (on l1 or on one particular l2), which stores the verification key for all wallets along with the rules for changing the key, and (ii) "wallet contracts" on l1 and many l2s, which read cross-chain to get the verification key. there are two ways to implement this: light version (check only to update keys): each wallet stores the verification key locally, and contains a function which can be called to check a cross-chain proof of the keystore's current state, and update its locally stored verification key to match. when a wallet is used for the first time on a particular l2, calling that function to obtain the current verification key from the keystore is mandatory. upside: uses cross-chain proofs sparingly, so it's okay if cross-chain proofs are expensive. all funds are only spendable with the current keys, so it's still secure. downside: to change the verification key, you have to make an on-chain key change in both the keystore and in every wallet that is already initialized (though not counterfactual ones). this could cost a lot of gas. heavy version (check for every tx): a cross-chain proof showing the key currently in the keystore is necessary for each transaction. upside: less systemic complexity, and keystore updating is cheap. downside: expensive per-tx, so requires much more engineering to make cross-chain proofs acceptably cheap. also not easily compatible with erc-4337, which currently does not support cross-contract reading of mutable objects during validation. what does a cross-chain proof look like? to show the full complexity, we'll explore the most difficult case: where the keystore is on one l2, and the wallet is on a different l2. if either the keystore on the wallet is on l1, then only half of this design is needed. let's assume that the keystore is on linea, and the wallet is on kakarot. a full proof of the keys to the wallet consists of: a proof proving the current linea state root, given the current ethereum state root that kakarot knows about a proof proving the current keys in the keystore, given the current linea state root there are two primary tricky implementation questions here: what kind of proof do we use? (is it merkle proofs? something else?) how does the l2 learn the recent l1 (ethereum) state root (or, as we shall see, potentially the full l1 state) in the first place? and alternatively, how does the l1 learn the l2 state root? in both cases, how long are the delays between something happening on one side, and that thing being provable to the other side? what kinds of proof schemes can we use? there are five major options: merkle proofs general-purpose zk-snarks special-purpose proofs (eg. with kzg) verkle proofs, which are somewhere between kzg and zk-snarks on both infrastructure workload and cost. no proofs and rely on direct state reading in terms of infrastructure work required and cost for users, i rank them roughly as follows: "aggregation" refers to the idea of aggregating all the proofs supplied by users within each block into a big meta-proof that combines all of them. this is possible for snarks, and for kzg, but not for merkle branches (you can combine merkle branches a little bit, but it only saves you log(txs per block) / log(total number of keystores), perhaps 15-30% in practice, so it's probably not worth the cost). aggregation only becomes worth it once the scheme has a substantial number of users, so realistically it's okay for a version-1 implementation to leave aggregation out, and implement that for version 2. how would merkle proofs work? this one is simple: follow the diagram in the previous section directly. more precisely, each "proof" (assuming the max-difficulty case of proving one l2 into another l2) would contain: a merkle branch proving the state-root of the keystore-holding l2, given the most recent state root of ethereum that the l2 knows about. the keystore-holding l2's state root is stored at a known storage slot of a known address (the contract on l1 representing the l2), and so the path through the tree could be hardcoded. a merkle branch proving the current verification keys, given the state-root of the keystore-holding l2. here once again, the verification key is stored at a known storage slot of a known address, so the path can be hardcoded. unfortunately, ethereum state proofs are complicated, but there exist libraries for verifying them, and if you use these libraries, this mechanism is not too complicated to implement. the larger problem is cost. merkle proofs are long, and patricia trees are unfortunately ~3.9x longer than necessary (precisely: an ideal merkle proof into a tree holding n objects is 32 * log2(n) bytes long, and because ethereum's patricia trees have 16 leaves per child, proofs for those trees are 32 * 15 * log16(n) ~= 125 * log2(n) bytes long). in a state with roughly 250 million (~2²⁸) accounts, this makes each proof 125 * 28 = 3500 bytes, or about 56,000 gas, plus extra costs for decoding and verifying hashes. two proofs together would end up costing around 100,000 to 150,000 gas (not including signature verification if this is used per-transaction) significantly more than the current base 21,000 gas per transaction. but the disparity gets worse if the proof is being verified on l2. computation inside an l2 is cheap, because computation is done off-chain and in an ecosystem with much fewer nodes than l1. data, on the other hand, has to be posted to l1. hence, the comparison is not 21000 gas vs 150,000 gas; it's 21,000 l2 gas vs 100,000 l1 gas. we can calculate what this means by looking at comparisons between l1 gas costs and l2 gas costs: l1 is currently about 15-25x more expensive than l2 for simple sends, and 20-50x more expensive for token swaps. simple sends are relatively data-heavy, but swaps are much more computationally heavy. hence, swaps are a better benchmark to approximate cost of l1 computation vs l2 computation. taking all this into account, if we assume a 30x cost ratio between l1 computation cost and l2 computation cost, this seems to imply that putting a merkle proof on l2 will cost the equivalent of perhaps fifty regular transactions. of course, using a binary merkle tree can cut costs by ~4x, but even still, the cost is in most cases going to be too high and if we're willing to make the sacrifice of no longer being compatible with ethereum's current hexary state tree, we might as well seek even better options. how would zk-snark proofs work? conceptually, the use of zk-snarks is also easy to understand: you simply replace the merkle proofs in the diagram above with a zk-snark proving that those merkle proofs exist. a zk-snark costs ~400,000 gas of computation, and about 400 bytes (compare: 21,000 gas and 100 bytes for a basic transaction, in the future reducible to ~25 bytes with compression). hence, from a computational perspective, a zk-snark costs 19x the cost of a basic transaction today, and from a data perspective, a zk-snark costs 4x as much as a basic transaction today, and 16x what a basic transaction may cost in the future. these numbers are a massive improvement over merkle proofs, but they are still quite expensive. there are two ways to improve on this: (i) special-purpose kzg proofs, or (ii) aggregation, similar to erc-4337 aggregation but using more fancy math. we can look into both. how would special-purpose kzg proofs work? warning, this section is much more mathy than other sections. this is because we're going beyond general-purpose tools and building something special-purpose to be cheaper, so we have to go "under the hood" a lot more. if you don't like deep math, skip straight to the next section. first, a recap of how kzg commitments work: we can represent a set of data [d_1 ... d_n] with a kzg proof of a polynomial derived from the data: specifically, the polynomial p where p(w) = d_1, p(w²) = d_2 ... p(wⁿ) = d_n. w here is a "root of unity", a value where wᴺ = 1 for some evaluation domain size n (this is all done in a finite field). to "commit" to p, we create an elliptic curve point com(p) = p₀ * g + p₁ * s₁ + ... + pₖ * sₖ. here: g is the generator point of the curve pᵢ is the i'th-degree coefficient of the polynomial p sᵢ is the i'th point in the trusted setup to prove p(z) = a, we create a quotient polynomial q = (p a) / (x z), and create a commitment com(q) to it. it is only possible to create such a polynomial if p(z) actually equals a. to verify a proof, we check the equation q * (x z) = p a by doing an elliptic curve check on the proof com(q) and the polynomial commitment com(p): we check e(com(q), com(x z)) ?= e(com(p) com(a), com(1)) some key properties that are important to understand are: a proof is just the com(q) value, which is 48 bytes com(p₁) + com(p₂) = com(p₁ + p₂) this also means that you can "edit" a value into an existing a commitment. suppose that we know that d_i is currently a, we want to set it to b, and the existing commitment to d is com(p). a commitment to "p, but with p(wⁱ) = b, and no other evaluations changed", then we set com(new_p) = com(p) + (b-a) * com(lᵢ), where lᵢ is a the "lagrange polynomial" that equals 1 at wⁱ and 0 at other wʲ points. to perform these updates efficiently, all n commitments to lagrange polynomials (com(lᵢ)) can be pre-calculated and stored by each client. inside a contract on-chain it may be too much to store all n commitments, so instead you could make a kzg commitment to the set of com(l_i) (or hash(com(l_i)) values, so whenever someone needs to update the tree on-chain they can simply provide the appropriate com(l_i) with a proof of its correctness. hence, we have a structure where we can just keep adding values to the end of an ever-growing list, though with a certain size limit (realistically, hundreds of millions could be viable). we then use that as our data structure to manage (i) a commitment to the list of keys on each l2, stored on that l2 and mirrored to l1, and (ii) a commitment to the list of l2 key-commitments, stored on the ethereum l1 and mirrored to each l2. keeping the commitments updated could either become part of core l2 logic, or it could be implemented without l2 core-protocol changes through deposit and withdraw bridges. a full proof would thus require: the latest com(key list) on the keystore-holding l2 (48 bytes) kzg proof of com(key list) being a value inside com(mirror_list), the commitment to the list of all key list comitments (48 bytes) kzg proof of your key in com(key list) (48 bytes, plus 4 bytes for the index) it's actually possible to merge the two kzg proofs into one, so we get a total size of only 100 bytes. note one subtlety: because the key list is a list, and not a key/value map like the state is, the key list will have to assign positions sequentially. the key commitment contract would contain its own internal registry mapping each keystore to an id, and for each key it would store hash(key, address of the keystore) instead of just key, to unambiguously communicate to other l2s which keystore a particular entry is talking about. the upside of this technique is that it performs very well on l2. the data is 100 bytes, ~4x shorter than a zk-snark and waaaay shorter than a merkle proof. the computation cost is largely one size-2 pairing check, or about 119,000 gas. on l1, data is less important than computation, and so unfortunately kzg is somewhat more expensive than merkle proofs. how would verkle trees work? verkle trees essentially involve stacking kzg commitments (or ipa commitments, which can be more efficient and use simpler cryptography) on top of each other: to store 2⁴⁸ values, you can make a kzg commitment to a list of 2²⁴ values, each of which itself is a kzg commitment to 2²⁴ values. verkle trees are being strongly considered for the ethereum state tree, because verkle trees can be used to hold key-value maps and not just lists (basically, you can make a size-2²⁵⁶ tree but start it empty, only filling in specific parts of the tree once you actually need to fill them). what a verkle tree looks like. in practice, you might give each node a width of 256 == 2⁸ for ipa-based trees, or 2²⁴ for kzg-based trees. proofs in verkle trees are somewhat longer than kzg; they might be a few hundred bytes long. they are also difficult to verify, especially if you try to aggregate many proofs into one. realistically, verkle trees should be considered to be like merkle trees, but more viable without snarking (because of the lower data costs), and cheaper with snarking (because of lower prover costs). the largest advantage of verkle trees is the possibility of harmonizing data structures: verkle proofs could be used directly over l1 or l2 state, without overlay structures, and using the exact same mechanism for l1 and l2. once quantum computers become an issue, or once proving merkle branches becomes efficient enough, verkle trees could be replaced in-place with a binary hash tree with a suitable snark-friendly hash function. aggregation if n users make n transactions (or more realistically, n erc-4337 useroperations) that need to prove n cross-chain claims, we can save a lot of gas by aggregating those proofs: the builder that would be combining those transactions into a block or bundle that goes into a block can create a single proof that proves all of those claims simultaneously. this could mean: a zk-snark proof of n merkle branches a kzg multi-proof a verkle multi-proof (or a zk-snark of a multi-proof) in all three cases, the proofs would only cost a few hundred thousand gas each. the builder would need to make one of these on each l2 for the users in that l2; hence, for this to be useful to build, the scheme as a whole needs to have enough usage that there are very often at least a few transactions within the same block on multiple major l2s. if zk-snarks are used, the main marginal cost is simply "business logic" of passing numbers around between contracts, so perhaps a few thousand l2 gas per user. if kzg multi-proofs are used, the prover would need to add 48 gas for each keystore-holding l2 that is used within that block, so the marginal cost of the scheme per user would add another ~800 l1 gas per l2 (not per user) on top. but these costs are much lower than the costs of not aggregating, which inevitably involve over 10,000 l1 gas and hundreds of thousands of l2 gas per user. for verkle trees, you can either use verkle multi-proofs directly, adding around 100-200 bytes per user, or you can make a zk-snark of a verkle multi-proof, which has similar costs to zk-snarks of merkle branches but is significantly cheaper to prove. from an implementation perspective, it's probably best to have bundlers aggregate cross-chain proofs through the erc-4337 account abstraction standard. erc-4337 already has a mechanism for builders to aggregate parts of useroperations in custom ways. there is even an implementation of this for bls signature aggregation, which could reduce gas costs on l2 by 1.5x to 3x depending on what other forms of compression are included. diagram from a bls wallet implementation post showing the workflow of bls aggregate signatures within an earlier version of erc-4337. the workflow of aggregating cross-chain proofs will likely look very similar. direct state reading a final possibility, and one only usable for l2 reading l1 (and not l1 reading l2), is to modify l2s to let them make static calls to contracts on l1 directly. this could be done with an opcode or a precompile, which allows calls into l1 where you provide the destination address, gas and calldata, and it returns the output, though because these calls are static-calls they cannot actually change any l1 state. l2s have to be aware of l1 already to process deposits, so there is nothing fundamental stopping such a thing from being implemented; it is mainly a technical implementation challenge (see: this rfp from optimism to support static calls into l1). notice that if the keystore is on l1, and l2s integrate l1 static-call functionality, then no proofs are required at all! however, if l2s don't integrate l1 static-calls, or if the keystore is on l2 (which it may eventually have to be, once l1 gets too expensive for users to use even a little bit), then proofs will be required. how does l2 learn the recent ethereum state root? all of the schemes above require the l2 to access either the recent l1 state root, or the entire recent l1 state. fortunately, all l2s have some functionality to access the recent l1 state already. this is because they need such a functionality to process messages coming in from l1 to the l2, most notably deposits. and indeed, if an l2 has a deposit feature, then you can use that l2 as-is to move l1 state roots into a contract on the l2: simply have a contract on l1 call the blockhash opcode, and pass it to l2 as a deposit message. the full block header can be received, and its state root extracted, on the l2 side. however, it would be much better for every l2 to have an explicit way to access either the full recent l1 state, or recent l1 state roots, directly. the main challenge with optimizing how l2s receive recent l1 state roots is simultaneously achieving safety and low latency: if l2s implement "direct reading of l1" functionality in a lazy way, only reading finalized l1 state roots, then the delay will normally be 15 minutes, but in the extreme case of inactivity leaks (which you have to tolerate), the delay could be several weeks. l2s absolutely can be designed to read much more recent l1 state roots, but because l1 can revert (even with single slot finality, reverts can happen during inactivity leaks), l2 would need to be able to revert as well. this is technically challenging from a software engineering perspective, but at least optimism already has this capability. if you use the deposit bridge to bring l1 state roots into l2, then simple economic viability might require a long time between deposit updates: if the full cost of a deposit is 100,000 gas, and we assume eth is at $1800, and fees are at 200 gwei, and l1 roots are brought into l2 once per day, that would be a cost of $36 per l2 per day, or $13148 per l2 per year to maintain the system. with a delay of one hour, that goes up to $315,569 per l2 per year. in the best case, a constant trickle of impatient wealthy users covers the updating fees and keep the system up to date for everyone else. in the worst case, some altruistic actor would have to pay for it themselves. "oracles" (at least, the kind of tech that some defi people call "oracles") are not an acceptable solution here: wallet key management is a very security-critical low-level functionality, and so it should depend on at most a few pieces of very simple, cryptographically trustless low-level infrastructure. additionally, in the opposite direction (l1s reading l2): on optimistic rollups, state roots take one week to reach l1 because of the fraud proof delay. on zk rollups it takes a few hours for now because of a combination of proving times and economic limits, though future technology will reduce this. pre-confirmations (from sequencers, attesters, etc) are not an acceptable solution for l1 reading l2. wallet management is a very security-critical low-level functionality, and so the level of security of the l2 -> l1 communication must be absolute: it should not even be possible to push a false l1 state root by taking over the l2 validator set. the only state roots the l1 should trust are state roots that have been accepted as final by the l2's state-root-holding contract on l1. some of these speeds for trustless cross-chain operations are unacceptably slow for many defi use cases; for those cases, you do need faster bridges with more imperfect security models. for the use case of updating wallet keys, however, longer delays are more acceptable: you're not delaying transactions by hours, you're delaying key changes. you'll just have to keep the old keys around longer. if you're changing keys because keys are stolen, then you do have a significant period of vulnerability, but this can be mitigated, eg. by wallets having a freeze function. ultimately, the best latency-minimizing solution is for l2s to implement direct reading of l1 state roots in an optimal way, where each l2 block (or the state root computation log) contains a pointer to the most recent l1 block, so if l1 reverts, l2 can revert as well. keystore contracts should be placed either on mainnet, or on l2s that are zk-rollups and so can quickly commit to l1. blocks of the l2 chain can have dependencies on not just previous l2 blocks, but also on an l1 block. if l1 reverts past such a link, the l2 reverts too. it's worth noting that this is also how an earlier (pre-dank) version of sharding was envisioned to work; see here for code. how much connection to ethereum does another chain need to hold wallets whose keystores are rooted on ethereum or an l2? surprisingly, not that much. it actually does not even need to be a rollup: if it's an l3, or a validium, then it's okay to hold wallets there, as long as you hold keystores either on l1 or on a zk rollup. the thing that you do need is for the chain to have direct access to ethereum state roots, and a technical and social commitment to be willing to reorg if ethereum reorgs, and hard fork if ethereum hard forks. one interesting research problem is identifying to what extent it is possible for a chain to have this form of connection to multiple other chains (eg. ethereum and zcash). doing it naively is possible: your chain could agree to reorg if ethereum or zcash reorg (and hard fork if ethereum or zcash hard fork), but then your node operators and your community more generally have double the technical and political dependencies. hence such a technique could be used to connect to a few other chains, but at increasing cost. schemes based on zk bridges have attractive technical properties, but they have the key weakness that they are not robust to 51% attacks or hard forks. there may be more clever solutions. preserving privacy ideally, we also want to preserve privacy. if you have many wallets that are managed by the same keystore, then we want to make sure: it's not publicly known that those wallets are all connected to each other. social recovery guardians don't learn what the addresses are that they are guarding. this creates a few issues: we cannot use merkle proofs directly, because they do not preserve privacy. if we use kzg or snarks, then the proof needs to provide a blinded version of the verification key, without revealing the location of the verification key. if we use aggregation, then the aggregator should not learn the location in plaintext; rather, the aggregator should receive blinded proofs, and have a way to aggregate those. we can't use the "light version" (use cross-chain proofs only to update keys), because it creates a privacy leak: if many wallets get updated at the same time due to an update procedure, the timing leaks the information that those wallets are likely related. so we have to use the "heavy version" (cross-chain proofs for each transaction). with snarks, the solutions are conceptually easy: proofs are information-hiding by default, and the aggregator needs to produce a recursive snark to prove the snarks. the main challenge of this approach today is that aggregation requires the aggregator to create a recursive snark, which is currently quite slow. with kzg, we can use this work on non-index-revealing kzg proofs (see also: a more formalized version of that work in the caulk paper) as a starting point. aggregation of blinded proofs, however, is an open problem that requires more attention. directly reading l1 from inside l2, unfortunately, does not preserve privacy, though implementing direct-reading functionality is still very useful, both to minimize latency and because of its utility for other applications. summary to have cross-chain social recovery wallets, the most realistic workflow is a wallet that maintains a keystore in one location, and wallets in many locations, where wallet reads the keystore either (i) to update their local view of the verification key, or (ii) during the process of verifying each transaction. a key ingredient of making this possible is cross-chain proofs. we need to optimize these proofs hard. either zk-snarks, waiting for verkle proofs, or a custom-built kzg solution, seem like the best options. in the longer term, aggregation protocols where bundlers generate aggregate proofs as part of creating a bundle of all the useroperations that have been submitted by users will be necessary to minimize costs. this should probably be integrated into the erc-4337 ecosystem, though changes to erc-4337 will likely be required. l2s should be optimized to minimize the latency of reading l1 state (or at least the state root) from inside the l2. l2s directly reading l1 state is ideal and can save on proof space. wallets can be not just on l2s; you can also put wallets on systems with lower levels of connection to ethereum (l3s, or even separate chains that only agree to include ethereum state roots and reorg or hard fork when ethereum reorgs or hardforks). however, keystores should be either on l1 or on high-security zk-rollup l2 . being on l1 saves a lot of complexity, though in the long run even that may be too expensive, hence the need for keystores on l2. preserving privacy will require additional work and make some options more difficult. however, we should probably move toward privacy-preserving solutions anyway, and at the least make sure that anything we propose is forward-compatible with preserving privacy. dark mode toggle the most important scarce resource is legitimacy 2021 mar 23 see all posts special thanks to karl floersch, aya miyaguchi and mr silly for ideas, feedback and review. the bitcoin and ethereum blockchain ecosystems both spend far more on network security the goal of proof of work mining than they do on everything else combined. the bitcoin blockchain has paid an average of about $38 million per day in block rewards to miners since the start of the year, plus about $5m/day in transaction fees. the ethereum blockchain comes in second, at $19.5m/day in block rewards plus $18m/day in tx fees. meanwhile, the ethereum foundation's annual budget, paying for research, protocol development, grants and all sorts of other expenses, is a mere $30 million per year. non-ef-sourced funding exists too, but it is at most only a few times larger. bitcoin ecosystem expenditures on r&d are likely even lower. bitcoin ecosystem r&d is largely funded by companies (with $250m total raised so far according to this page), and this report suggests about 57 employees; assuming fairly high salaries and many paid developers not being counted, that works out to about $20m per year. clearly, this expenditure pattern is a massive misallocation of resources. the last 20% of network hashpower provides vastly less value to the ecosystem than those same resources would if they had gone into research and core protocol development. so why not just.... cut the pow budget by 20% and redirect the funds to those other things instead? the standard answer to this puzzle has to do with concepts like "public choice theory" and "schelling fences": even though we could easily identify some valuable public goods to redirect some funding to as a one-off, making a regular institutionalized pattern of such decisions carries risks of political chaos and capture that are in the long run not worth it. but regardless of the reasons why, we are faced with this interesting fact that the organisms that are the bitcoin and ethereum ecosystems are capable of summoning up billions of dollars of capital, but have strange and hard-to-understand restrictions on where that capital can go. the powerful social force that is creating this effect is worth understanding. as we are going to see, it's also the same social force behind why the ethereum ecosystem is capable of summoning up these resources in the first place (and the technologically near-identical ethereum classic is not). it's also a social force that is key to helping a chain recover from a 51% attack. and it's a social force that underlies all sorts of extremely powerful mechanisms far beyond the blockchain space. for reasons that will be clear in the upcoming sections, i will give this powerful social force a name: legitimacy. coins can be owned by social contracts to better understand the force that we are getting at, another important example is the epic saga of steem and hive. in early 2020, justin sun bought steem-the-company, which is not the same thing as steem-the-blockchain but did hold about 20% of the steem token supply. the community, naturally, did not trust justin sun. so they made an on-chain vote to formalize what they considered to be a longstanding "gentleman's agreement" that steem-the-company's coins were held in trust for the common good of steem-the-blockchain and should not be used to vote. with the help of coins held by exchanges, justin sun made a counterattack, and won control of enough delegates to unilaterally control the chain. the community saw no further in-protocol options. so instead they made a fork of steem-the-blockchain, called hive, and copied over all of the steem token balances except those, including justin sun's, which participated in the attack. and they got plenty of applications on board. if they had not managed this, far more users would have either stayed on steem or moved to some different project entirely. the lesson that we can learn from this situation is this: steem-the-company never actually "owned" the coins. if they did, they would have had the practical ability to use, enjoy and abuse the coins in whatever way they wanted. but in reality, when the company tried to enjoy and abuse the coins in a way that the community did not like, they were successfully stopped. what's going on here is a pattern of a similar type to what we saw with the not-yet-issued bitcoin and ethereum coin rewards: the coins were ultimately owned not by a cryptographic key, but by some kind of social contract. we can apply the same reasoning to many other structures in the blockchain space. consider, for example, the ens root multisig. the root multisig is controlled by seven prominent ens and ethereum community members. but what would happen if four of them were to come together and "upgrade" the registrar to one that transfers all the best domains to themselves? within the context of ens-the-smart-contract-system, they have the complete and unchallengeable ability to do this. but if they actually tried to abuse their technical ability in this way, what would happen is clear to anyone: they would be ostracized from the community, the remaining ens community members would make a new ens contract that restores the original domain owners, and every ethereum application that uses ens would repoint their ui to use the new one. this goes well beyond smart contract structures. why is it that elon musk can sell an nft of elon musk's tweet, but jeff bezos would have a much harder time doing the same? elon and jeff have the same level of ability to screenshot elon's tweet and stick it into an nft dapp, so what's the difference? to anyone who has even a basic intuitive understanding of human social psychology (or the fake art scene), the answer is obvious: elon selling elon's tweet is the real thing, and jeff doing the same is not. once again, millions of dollars of value are being controlled and allocated, not by individuals or cryptographic keys, but by social conceptions of legitimacy. and, going even further out, legitimacy governs all sorts of social status games, intellectual discourse, language, property rights, political systems and national borders. even blockchain consensus works the same way: the only difference between a soft fork that gets accepted by the community and a 51% censorship attack after which the community coordinates an extra-protocol recovery fork to take out the attacker is legitimacy. so what is legitimacy? see also: my earlier post on blockchain governance. to understand the workings of legitimacy, we need to dig down into some game theory. there are many situations in life that demand coordinated behavior: if you act in a certain way alone, you are likely to get nowhere (or worse), but if everyone acts together a desired result can be achieved. an abstract coordination game. you benefit heavily from making the same move as everyone else. one natural example is driving on the left vs right side of the road: it doesn't really matter what side of the road people drive on, as long as they drive on the same side. if you switch sides at the same time as everyone else, and most people prefer the new arrangement, there can be a net benefit. but if you switch sides alone, no matter how much you prefer driving on the other side, the net result for you will be quite negative. now, we are ready to define legitimacy. legitimacy is a pattern of higher-order acceptance. an outcome in some social context is legitimate if the people in that social context broadly accept and play their part in enacting that outcome, and each individual person does so because they expect everyone else to do the same. legitimacy is a phenomenon that arises naturally in coordination games. if you're not in a coordination game, there's no reason to act according to your expectation of how other people will act, and so legitimacy is not important. but as we have seen, coordination games are everywhere in society, and so legitimacy turns out to be quite important indeed. in almost any environment with coordination games that exists for long enough, there inevitably emerge some mechanisms that can choose which decision to take. these mechanisms are powered by an established culture that everyone pays attention to these mechanisms and (usually) does what they say. each person reasons that because everyone else follows these mechanisms, if they do something different they will only create conflict and suffer, or at least be left in a lonely forked ecosystem all by themselves. if a mechanism successfully has the ability to make these choices, then that mechanism has legitimacy. a byzantine general rallying his troops forward. the purpose of this isn't just to make the soldiers feel brave and excited, but also to reassure them that everyone else feels brave and excited and will charge forward as well, so an individual soldier is not just committing suicide by charging forward alone. in any context where there's a coordination game that has existed for long enough, there's likely a conception of legitimacy. and blockchains are full of coordination games. which client software do you run? which decentralized domain name registry do you ask for which address corresponds to a .eth name? which copy of the uniswap contract do you accept as being "the" uniswap exchange? even nfts are a coordination game. the two largest parts of an nft's value are (i) pride in holding the nft and ability to show off your ownership, and (ii) the possibility of selling it in the future. for both of these components, it's really really important that whatever nft you buy is recognized as legitimate by everyone else. in all of these cases, there's a great benefit to having the same answer as everyone else, and the mechanism that determines that equilibrium has a lot of power. theories of legitimacy there are many different ways in which legitimacy can come about. in general, legitimacy arises because the thing that gains legitimacy is psychologically appealing to most people. but of course, people's psychological intuitions can be quite complex. it is impossible to make a full listing of theories of legitimacy, but we can start with a few: legitimacy by brute force: someone convinces everyone that they are powerful enough to impose their will and resisting them will be very hard. this drives most people to submit because each person expects that everyone else will be too scared to resist as well. legitimacy by continuity: if something was legitimate at time t, it is by default legitimate at time t+1. legitimacy by fairness: something can become legitimate because it satisfies an intuitive notion of fairness. see also: my post on credible neutrality, though note that this is not the only kind of fairness. legitimacy by process: if a process is legitimate, the outputs of that process gain legitimacy (eg. laws passed by democracies are sometimes described in this way). legitimacy by performance: if the outputs of a process lead to results that satisfy people, then that process can gain legitimacy (eg. successful dictatorships are sometimes described in this way). legitimacy by participation: if people participate in choosing an outcome, they are more likely to consider it legitimate. this is similar to fairness, but not quite: it rests on a psychological desire to be consistent with your previous actions. note that legitimacy is a descriptive concept; something can be legitimate even if you personally think that it is horrible. that said, if enough people think that an outcome is horrible, there is a higher chance that some event will happen in the future that will cause that legitimacy to go away, often at first gradually, then suddenly. legitimacy is a powerful social technology, and we should use it the public goods funding situation in cryptocurrency ecosystems is fairly poor. there are hundreds of billions of dollars of capital flowing around, but public goods that are key to that capital's ongoing survival are receiving only tens of millions of dollars per year of funding. there are two ways to respond to this fact. the first way is to be proud of these limitations and the valiant, even if not particularly effective, efforts that your community makes to work around them. this seems to be the route that the bitcoin ecosystem often takes: the personal self-sacrifice of the teams funding core development is of course admirable, but it's admirable the same way that eliud kipchoge running a marathon in under 2 hours is admirable: it's an impressive show of human fortitude, but it's not the future of transportation (or, in this case, public goods funding). much like we have much better technologies to allow people to move 42 km in under an hour without exceptional fortitude and years of training, we should also focus on building better social technologies to fund public goods at the scales that we need, and as a systemic part of our economic ecology and not one-off acts of philanthropic initiative. now, let us get back to cryptocurrency. a major power of cryptocurrency (and other digital assets such as domain names, virtual land and nfts) is that it allows communities to summon up large amounts of capital without any individual person needing to personally donate that capital. however, this capital is constrained by conceptions of legitimacy: you cannot simply allocate it to a centralized team without compromising on what makes it valuable. while bitcoin and ethereum do already rely on conceptions of legitimacy to respond to 51% attacks, using conceptions of legitimacy to guide in-protocol funding of public goods is much harder. but at the increasingly rich application layer where new protocols are constantly being created, we have quite a bit more flexibility in where that funding could go. legitimacy in bitshares one of the long-forgotten, but in my opinion very innovative, ideas from the early cryptocurrency space was the bitshares social consensus model. essentially, bitshares described itself as a community of people (pts and ags holders) who were willing to help collectively support an ecosystem of new projects, but for a project to be welcomed into the ecosystem, it would have to allocate 10% of its token supply to existing pts and ags holders. now, of course anyone can make a project that does not allocate any coins to pts/ags holders, or even fork a project that did make an allocation and take the allocation out. but, as dan larimer says: you cannot force anyone to do anything, but in this market is is all network effect. if someone comes up with a compelling implementation then you can adopt the entire pts community for the cost of generating a new genesis block. the individual who decided to start from scratch would have to build an entire new community around his system. considering the network effect, i suspect that the coin that honors protoshares will win. this is also a conception of legitimacy: any project that makes the allocation to pts/ags holders will get the attention and support of the community (and it will be worthwhile for each individual community member to take an interest in the project because the rest of the community is doing so as well), and any project that does not make the allocation will not. now, this is certainly not a conception of legitimacy that we want to replicate verbatim there is little appetite in the ethereum community for enriching a small group of early adopters but the core concept can be adapted into something much more socially valuable. extending the model to ethereum blockchain ecosystems, ethereum included, value freedom and decentralization. but the public goods ecology of most of these blockchains is, regrettably, still quite authority-driven and centralized: whether it's ethereum, zcash or any other major blockchain, there is typically one (or at most 2-3) entities that far outspend everyone else, giving independent teams that want to build public goods few options. i call this model of public goods funding "central capital coordinators for public-goods" (cccps). this state of affairs is not the fault of the organizations themselves, who are typically valiantly doing their best to support the ecosystem. rather, it's the rules of the ecosystem that are being unfair to that organization, because they hold the organization to an unfairly high standard. any single centralized organization will inevitably have blindspots and at least a few categories and teams whose value that it fails to understand; this is not because anyone involved is doing anything wrong, but because such perfection is beyond the reach of small groups of humans. so there is great value in creating a more diversified and resilient approach to public goods funding to take the pressure off any single organization. fortunately, we already have the seed of such an alternative! the ethereum application-layer ecosystem exists, is growing increasingly powerful, and is already showing its public-spiritedness. companies like gnosis have been contributing to ethereum client development, and various ethereum defi projects have donated hundreds of thousands of dollars to the gitcoin grants matching pool. gitcoin grants has already achieved a high level of legitimacy: its public goods funding mechanism, quadratic funding, has proven itself to be credibly neutral and effective at reflecting the community's priorities and values and plugging the holes left by existing funding mechanisms. sometimes, top gitcoin grants matching recipients are even used as inspiration for grants by other and more centralized grant-giving entities. the ethereum foundation itself has played a key role in supporting this experimentation and diversity, incubating efforts like gitcoin grants, along with molochdao and others, that then go on to get broader community support. we can make this nascent public goods-funding ecosystem even stronger by taking the bitshares model, and making a modification: instead of giving the strongest community support to projects who allocate tokens to a small oligarchy who bought pts or ags back in 2013, we support projects that contribute a small portion of their treasuries toward the public goods that make them and the ecosystem that they depend on possible. and, crucially, we can deny these benefits to projects that fork an existing project and do not give back value to the broader ecosystem. there are many ways to do support public goods: making a long-term commitment to support the gitcoin grants matching pool, supporting ethereum client development (also a reasonably credibly-neutral task as there's a clear definition of what an ethereum client is), or even running one's own grant program whose scope goes beyond that particular application-layer project itself. the easiest way to agree on what counts as sufficient support is to agree on how much for example, 5% of a project's spending going to support the broader ecosystem and another 1% going to public goods that go beyond the blockchain space and rely on good faith to choose where that funding would go. does the community actually have that much leverage? of course, there are limits to the value of this kind of community support. if a competing project (or even a fork of an existing project) gives its users a much better offering, then users are going to flock to it, regardless of how many people yell at them to instead use some alternative that they consider to be more pro-social. but these limits are different in different contexts; sometimes the community's leverage is weak, but at other times it's quite strong. an interesting case study in this regard is the case of tether vs dai. tether has many scandals, but despite this traders use tether to hold and move around dollars all the time. the more decentralized and transparent dai, despite its benefits, is unable to take away much of tether's market share, at least as far as traders go. but where dai excels is applications: augur uses dai, xdai uses dai, pooltogether uses dai, zk.money plans to use dai, and the list goes on. what dapps use usdt? far fewer. hence, though the power of community-driven legitimacy effects is not infinite, there is nevertheless considerable room for leverage, enough to encourage projects to direct at least a few percent of their budgets to the broader ecosystem. there's even a selfish reason to participate in this equilibrium: if you were the developer of an ethereum wallet, or an author of a podcast or newsletter, and you saw two competing projects, one of which contributes significantly to ecosystem-level public goods including yourself and one which does not, for which one would you do your utmost to help them secure more market share? nfts: supporting public goods beyond ethereum the concept of supporting public goods through value generated "out of the ether" by publicly supported conceptions of legitimacy has value going far beyond the ethereum ecosystem. an important and immediate challenge and opportunity is nfts. nfts stand a great chance of significantly helping many kinds of public goods, especially of the creative variety, at least partially solve their chronic and systemic funding deficiencies. actually a very admirable first step. but they could also be a missed opportunity: there is little social value in helping elon musk earn yet another $1 million by selling his tweet when, as far as we can tell, the money is just going to himself (and, to his credit, he eventually decided not to sell). if nfts simply become a casino that largely benefits already-wealthy celebrities, that would be a far less interesting outcome. fortunately, we have the ability to help shape the outcome. which nfts people find attractive to buy, and which ones they do not, is a question of legitimacy: if everyone agrees that one nft is interesting and another nft is lame, then people will strongly prefer buying the first, because it would have both higher value for bragging rights and personal pride in holding it, and because it could be resold for more because everyone else is thinking in the same way. if the conception of legitimacy for nfts can be pulled in a good direction, there is an opportunity to establish a solid channel of funding to artists, charities and others. here are two potential ideas: some institution (or even dao) could "bless" nfts in exchange for a guarantee that some portion of the revenues goes toward a charitable cause, ensuring that multiple groups benefit at the same time. this blessing could even come with an official categorization: is the nft dedicated to global poverty relief, scientific research, creative arts, local journalism, open source software development, empowering marginalized communities, or something else? we can work with social media platforms to make nfts more visible on people's profiles, giving buyers a way to show the values that they committed not just their words but their hard-earned money to. this could be combined with (1) to nudge users toward nfts that contribute to valuable social causes. there are definitely more ideas, but this is an area that certainly deserves more active coordination and thought. in summary the concept of legitimacy (higher-order acceptance) is very powerful. legitimacy appears in any context where there is coordination, and especially on the internet, coordination is everywhere. there are different ways in which legitimacy comes to be: brute force, continuity, fairness, process, performance and participation are among the important ones. cryptocurrency is powerful because it lets us summon up large pools of capital by collective economic will, and these pools of capital are, at the beginning, not controlled by any person. rather, these pools of capital are controlled directly by concepts of legitimacy. it's too risky to start doing public goods funding by printing tokens at the base layer. fortunately, however, ethereum has a very rich application-layer ecosystem, where we have much more flexibility. this is in part because there's an opportunity not just to influence existing projects, but also shape new ones that will come into existence in the future. application-layer projects that support public goods in the community should get the support of the community, and this is a big deal. the example of dai shows that this support really matters! the etherem ecosystem cares about mechanism design and innovating at the social layer. the ethereum ecosystem's own public goods funding challenges are a great place to start! but this goes far beyond just ethereum itself. nfts are one example of a large pool of capital that depends on concepts of legitimacy. the nft industry could be a significant boon to artists, charities and other public goods providers far beyond our own virtual corner of the world, but this outcome is not predetermined; it depends on active coordination and support. stateful turing-complete policies | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search stateful turing-complete policies posted by vitalik buterin on november 9, 2015 research & development one of the major security challenges of the internet over the last twenty years has consistently been the rather simple problem of securing user accounts. right now, users have accounts with hundreds of websites, and dozens of passwords, leading to large numbers of hacks as individual websites, often run by people not particularly skilled in the finer points of cryptography and internet security, find themselves exploited by increasingly clever hackers, and users frequently deal with the complexity of remembering hundreds of passwords by either making them simple or making them all the same with often very unfortunate results. over time, a patchwork of ad-hoc solutions has certainly developed, including the use of one's email account as a universal backup, and "password manager" software like lastpass, though at high cost: such solutions either retain much of the underlying complexity of password-bsaed access or give centralized companies very high degrees of control over your online life. there are many calls to get rid of passwords, but the question is: what do we replace them with? there are many ideas, ranging from "one single password to rule them all" to smartphone authentication to specialized hardware devices and biometrics and all sorts of multi-factor m-of-n policies, but even these more complex constructions so far have typically been application-specific: many banks now give you a specialized access device to log into your bank account, but if you trust its security you cannot also use it to access your email. in general, we see that the problem of how to best manage user access control and minimize key loss and theft risks is complex enough that it never will be solved "once and for all", and so the best way to solve it is to allow a free market of solutions to flourish and let each user pick which ones work best for them; however, the way to make that actually happen is by unbundling the "access control solutions" market from the "services" market. that is to say, exactly what we are to a large extent not doing right now. the hardware access device to my ubs bank account. remind me, why can't i also use this to secure my domains on namecheap? so how do we do that? the first step is to introduce some well-placed use of the ultimate abstraction: turing-complete code. rather than, at the protocol level, allowing users to specify a password, or providing a pre-selected set of providers, or even a standard which relies on talking to a server of the user's choice, allow access policies to be specified in code to be executed in a deterministic virtual machine (where the evm is a good a start as any). code can include digital signature verifications using any cryptographic algorithm (so you get forward-compatibility with quantum-safe crypto for free), potentially including keys held on the user's computer, keys directly derived from a password, keys held on a hardware device or any arbitrary policy including any combination of the above. this way, innovation can happen in access-control mechanisms without any need for websites (or other systems requiring authentication) to do anything to accomodate new changes. additionally, the system neatly allows organizations to use the scheme using multi-person access controls right away, without any further need for integration. the next step is turing-complete operation-dependent code. for many applications, you want the ability to authorize some users to carry out some operations but not others; for example, you may want to authorize a sysadmin to change the ip address that a domain name points to, but not sell the domain outright. to accomodate this, the abstraction needs to change. a simple "turing-complete-code as signature" setup might have the following form: vm(code, server-provided nonce ++ signature) ?= 1 where vm is a virtual machine that runs code, taking a server-provided nonce and a signature as input, and the verification check is to see whether or not the output is 1. a simple example of code that could be put in is an elliptic curve digital signature verifier. to allow different authorization requirements depending on the operation, you want: vm(code, server-provided nonce ++ operation_data ++ signature) ?= 1 a signature would need to be provided with every operation that the user wants to carry out (this has the benefit of providing definite, third-party-verifiable, proof that an operation was authorized); the operation data (imagine the function name and the arguments encoded in an ethereum-style abi) would be added as an argument for the virtual machine, and the signature would have to be over both the nonce and the operation data. this gets you quite far, but in some cases not far enough. one simple example is this: what if you want to give someone permission to withdraw small amounts of money but not large amounts, ie. a withdrawal limit? in that case, the problem that you must overcome is simple: what if someone limited by a withdrawal cap of $100 tries to evade it by simply running a script to withdraw $90 over and over again? to solve this, you need a smarter withdrawal limit; essentially, something like "maximum $100 per day". another natural case is key revocation: if a key gets hacked or lost, you want to replace it, and you want to make sure that the world finds out that your policy was changed so that attackers cannot try to impersonate you under your old policy. to get past this last hump, we need to go one step further: we need turing-complete operation-dependent stateful policies; that is to say, operations should be able to change the state of the policy. and here is where not just cryptography, but specifically blockchains come in. of course, you could just have a central server manage the whole thing, and many people are perfectly fine with trusting a central server, but blockchains are moderately valuable here because they are more convenient, provide a credible story of neutrality, and are easier to standardize around. ultimately, as it would be quite harmful for innovation to permanently choose "one blockchain to rule them all", the thing that we want to standardize is a mechanism by which users can download modules to support any blockchain or centralized solution as they wish. for blockchain-based applications, having a stateful policy enforced right on the blockchain makes natural sense; there is no need to involve yet another special class of intermediaries, and people can start doing it right now. the abstraction of an "account" that ethereum offers makes it extremely easy to work with this approach: if your application works with simple users holding private keys, it also works for just about every kind of individual, multiparty, hardware-driven, military-grade or whatever other policy users will come up with in the future. for other applications, users may want privacy, both in the state-changing operations that they perform and even in the nature of their policy at any one particular time. for this reason, you likely want a solution like hawk, where the blockchain still ensures the security of the process but, thanks to the wonders of zero-knowledge-proof technology, knows nothing about what is being secured; before hawk is implemented, simpler forms of cryptography such as ring signatures may suffice. other applications account security is the first, and most basic, application for the concept of code as policy, there are also others. one simple one is a domain name registry. onename, one of the popular "decentralized name registry" services, is currently planning on implementing a feature where top-level domains can choose fee policies for subdomains based on the number of letters, consonants and vowels. this is useful, but of course economically ugly: there are definitely hundreds of characteristics other than letters, consonants and vowels that can influence a domain name price, and people may even want to experiment with other registration strategies like different types of auctions. once again, an even nicer solution is to apply some simple modularity: let people create their own namespace in stateful turing-complete code. if you are doing this on a platform where stateful turing-complete code exists, you can just allow an address to control a subdomain, and then, tada, you support stateful turing-complete subdomain policies already. this is the essence of object-oriented programming: expose an interface, and allow other objects, which can have arbitrarily complex internal code, satisfy that interface. a further one is private stock trading. particularly in the case of privately held companies, stock trading is not, and cannot, be completely free and unrestricted the way that trading of cryptocurrencies is; companies often want to have restrictions such as: giving employees shares and allowing them to sell them only after some period of time requiring new shareholders to be approved by existing shareholders, with the possibility of such approvals coming with a cap on how many shares can be owned by that specific holder forced-buyout procedures restricting the maximum rate at which stocks are sold (ie. withdrawal limits) or requiring waiting periods or offering specific other holders right of first refusal sure, you can create a private blockchain-based stock trading platform for one client, and provide the restrictions that that one client wants. but what if other clients want different restrictions? you may as well nip the problem in the bud, at least at the "core application layer", and solve it once and for all by... allowing each individual stock, represented as a sub-currency, to have restrictions represented as stateful turing-complete code. this functionality can be represented in the "token" api by extending it, for example, as follows: getminimumbalance(account): get the minimum balance that an account can hold at the current time getmaximumbalance(account): get the maximum balance that an account can hold at the current time in short, applications don't have policies; applications interact with objects (user accounts, currencies, etc), and objects have policies. alternatively, even shorter: are you building a blockchain-based financial derivatives application, and someone is asking you to add a feature to allow a vote between multiple data feed providers instead of just one? don't even think about it; instead, just set one data feed provider address, and allow users to come up with their own policies; the upside is that whatever code they use or write, they'll be able to use to more securely provide data feeds for the arbitration dapp as well. are you building a dns system, and someone is asking you to introduce support for specific auction types for subdomains? don't to it at the root dns level; instead, allow subdomains to be addresses, and allow users to invent their own auction algorithms; whatever algorithms they invent, they'll be able to use for their registry for decentralized chat usernames as well. this is the benefit of abstraction: account security policy design can become a self-contained field of study to itself, and whatever new solutions exist can instantly be applied everywhere. some people will want to trust a third party; others will want to have a multi-signature authorization between five of their own different devices, and some will want a key to themselves with the option for three of five friends to come together to reset the key to a new one. some will want an access policy where, if they make no transactions within twelve months, they are presumed dead and a lawyer will gain access in order to be able to execute on their will for all of their digital assets. and some will want a policy which gives one key full control for applications that declare themselves low-security but two of three keys for applications that declare themselves high-security. name registry pricing policy design can become self-contained as well as can digital asset ownership restriction policy, a field that would interest everyone from small and large traditional corporations to community-based daos. and that is the power of a stateful turing-complete code. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements security alert [consensus issue] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security alert [consensus issue] posted by jutta steiner on august 20, 2015 security this alert is related to a consensus issue that occurred on the frontier network at block 116,522, mined on 2015-08-20 at 14:59:16+02:00 – issue has been fixed, see "fix" below. impact: high issue description: state database consensus issue in geth with deletion of account data, which could happen during suicide instructions. affected implementations: all geth implementations up to and including versions 1.0.1, 1.1.0 and develop (“unstable 1.1.0”) were affected. eth (c++) and pyethereum (python) are unaffected. effects on expected chain reorganisation depth: increase waiting time for eventual block confirmation to 12 hours proposed temporary workaround: miners switch to eth or pyethereum asap remedial action taken by ethereum: provision of fixes as below. fix: note that the consensus  issue occured just before the announcement of the new release 1.1.0. when upgrading, please make sure you upgrade to the version you intended as you might not want to fix and upgrade from 1.0.1 to 1.1.0 (which has not yet been officially released) at the same time. fixes below are for version 1.0.2 – builds are generated for v 1.0.2.  release 1.0.2 including source and binaries can be found here if you are building from source: git pull followed by make geth – please use the master branch commit a0303ff4bdc17fba10baea4ce9ff250e5923efa2 if using the ppa: sudo apt-get update then sudo apt-get upgrade we are still working on the brew fix the correct version for this update on ubuntu and osx is geth/v1.0.2-a0303f note that it is likely that you receive the following message alert which will resolve itself after a while once your peers have updated their clients: i0820 19:00:53.368852    4539 chain_manager.go:776] bad block #116522 (05bef30ef572270f654746da22639a7a0c97dd97a7050b9e252391996aaeb689) i0820 19:00:53.368891    4539 chain_manager.go:777] found known bad hash in chain 05bef30ef572270f654746da22639a7a0c97dd97a7050b9e updates: 20/08/15 17:47+02:00 issue has been identified and fix for geth is underway. geth(go)-miners should upgrade asap once the fixes become available and meanwhile switch to eth (c++) or python. eth (c++) and python miners are unaffected. 20/08/15 19:38+02:00 official release of fixes, see above. 20/08/15 21:19+02:00 note that the consenus  issue occured just before the announcement of the new release 1.0.2. therefore, please make sure that you get the fix you want, i.e. remain on 1.0.1 or upgrade to 1.0.2. see "fix" for more details. 21/08/15 11:30+2:00 since the occurrence of the consensus issue, we've been closely monitoring the chain and network. according to network statistics, most miners have upgraded to the patched 1.0.2 or switched to eth. our expectations concerning maximum likely chain-reorganization depth is 750 blocks. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle gitcoin grants round 5 retrospective 2020 apr 30 see all posts special thanks to kevin owocki and frank chen for help and review round 5 of gitcoin grants has just finished, with $250,000 of matching split between tech, media, and the new (non-ethereum-centric) category of "public health". in general, it seems like the mechanism and the community are settling down into a regular rhythm. people know what it means to contribute, people know what to expect, and the results emerge in a relatively predictable pattern even if which specific grants get the most funds is not so easy to predict. stability of income so let's go straight into the analysis. one important property worth looking at is stability of income across rounds: do projects that do well in round n also tend to do well in round n+1? stability of income is very important if we want to support an ecosystem of "quadratic freelancers": we want people to feel comfortable relying on their income knowing that it will not completely disappear the next round. on the other hand, it would be harmful if some recipients became completely entrenched, with no opportunity for new projects to come in and compete for the pot, so there is a need for a balance. on the media side, we do see some balance between stability and dynamism: week in ethereum had the highest total amount received in both the previous round and the current round. ethhub and bankless are also near the top in both the current round and the previous round. on the other hand, antiprosynthesis, the (beloved? notorious? famous?) twitter info-warrior, has decreased from $13,813 to $5,350, while chris blec's youtube channel has increased from $5,851 to $12,803. so some churn, but also some continuity between rounds. on the tech side, we see much more churn in the winners, with a less clear relationship between income last round and income this round: last round, the winner was tornado cash, claiming $30,783; this round, they are down to $8,154. this round, the three roughly-even winners are samczsun ($4,631 contributions + $15,704 match = $20,335 total), arboreum ($16,084 contributions + $9,046 match = $25,128 total) and 1inch.exchange ($58,566 contributions + $7,893 match = $66,459 total), in the latter case the bulk coming from one contribution: in the previous round, those three winners were not even in the top ten, and in some cases not even part of gitcoin grants at all. these numbers show us two things. first, large parts of the gitcoin community seem to be in the mindset of treating grants not as a question of "how much do you deserve for your last two months of work?", but rather as a one-off reward for years of contributions in the past. this was one of the strongest rebuttals that i received to my criticism of antiprosynthesis receiving $13,813 in the last round: that the people who contributed to that award did not see it as two months' salary, but rather as a reward for years of dedication and work for the ethereum ecosystem. in the next round, contributors were content that the debt was sufficiently repaid, and so they moved on to give a similar gift of appreciation and gratitude to chris blec. that said, not everyone contributes in this way. for example, prysm got $7,966 last round and $8,033 this round, and week in ethereum is consistently well-rewarded ($16,727 previous, $12,195 current), and ethhub saw less stability but still kept half its income ($13,515 previous, $6,705 current) even amid a 20% drop to the matching pool size as some funds were redirected to public health. so there definitely are some contributors that are getting almost a reasonable monthly salary from gitcoin grants (yes, even these amounts are all serious underpayment, but remember that the pool of funds gitcoin grants has to distribute in the first place is quite small, so there's no allocation that would not seriously underpay most people; the hope is that in the future we will find ways to make the matching pot grow bigger). why didn't more people use recurring contributions? one feature that was tested this round to try to improve stability was recurring contributions: users could choose to split their contribution among multiple rounds. however, the feature was not used often: out of over 8,000 total contributions, only 120 actually made recurring contributions. i can think of three possible explanations for this: people just don't want to give recurring contributions; they genuinely prefer to freshly rethink who they are supporting every round. people would be willing to give more recurring contributions, but there is some kind of "market failure" stopping them; that is, it's collectively optimal for everyone to give more recurring contributions, but it's not any individual contributor's interest to be the first to do so. there's some ui inconveniences or other "incidental" obstacles preventing recurring contributions. in a recent call with the gitcoin team, hypothesis (3) was mentioned frequently. a specific issue was that people were worried about making recurring contributions because they were concerned whether or not the money that they lock up for a recurring contribution would be safe. improving the payment system and notification workflow may help with this. another option is to move away from explicit "streaming" and instead simply have the ui provide an option for repeating the last round's contributions and making edits from there. hypothesis (1) also should be taken seriously; there's genuine value in preventing ossification and allowing space for new entrants. but i want to zoom in particularly on hypothesis (2), the coordination failure hypothesis. my explanation of hypothesis (2) starts, interestingly enough, with a defense of (1): why ossification is genuinely a risk. suppose that there are two projects, a and b, and suppose that they are equal quality. but a already has an established base of contributors; b does not (we'll say for illustration it only has a few existing contributors). here's how much matching you are contributing by participating in each project: contributing to a contributing to b clearly, you have more impact by supporting a, and so a gets even more contributors and b gets fewer; the rich get richer. even if project b was somewhat better, the greater impact from supporting a could still create a lock-in that reinforces a's position. the current everyone-starts-from-zero-in-each-round mechanism greatly limits this type of entrenchment, because, well, everyone's matching gets reset and starts from zero. however, a very similar effect also is the cause behind the market failure preventing stable recurring contributions, and the every-round-reset actually exacerbates it. look at the same picture above, except instead of thinking of a and b as two different projects, think of them as the same project in the current round and in the next round. we simplify the model as follows. an individual has two choices: contribute $10 in the current round, or contribute $5 in the current round and $5 in the next round. if the matchings in the two rounds were equal, then the latter option would actually be more favorable: because the matching is proportional to the square root of the donation size, the former might give you eg. a $200 match now, but the latter would give you $141 in the current round + $141 in the next round = $282. but if you see a large mass of people contributing in the current round, and you expect much fewer people to contribute in the second round, then the choice is not $200 versus $141 + $141, it might be $200 versus $141 + $5. and so you're better off joining the current round's frenzy. we can mathematically analyze the equilibrium: so there is a substantial region within which the bad equilibrium of everyone concentrating is sticky: if more than about 3/4 of contributors are expected to concentrate, it seems in your interest to also concentrate. a mathematically astute reader may note that there is always some intermediate strategy that involves splitting but at a ratio different from 50/50, which you can prove performs better than either full concentrating or the even split, but here we get back to hypothesis (3) above: the ui doesn't offer such a complex menu of choices, it just offers the choice of a one-time contribution or a recurring contribution, so people pick one or the other. how might we fix this? one option is to add a bit of continuity to matching ratios: when computing pairwise matches, match against not just the current round's contributors but, say, 1/3 of the previous round's contributors as well: this makes some philosophical sense: the objective of quadratic funding is to subsidize contributions to projects that are detected to be public goods because multiple people have contributed to them, and contributions in the previous round are certainly also evidence of a project's value, so why not reuse those? so here, moving away from everyone-starts-from-zero toward this partial carryover of matching ratios would mitigate the round concentration effect but, of course, it would exacerbate the risk of entrenchment. hence, some experimentation and balance may be in order. a broader philosophical question is, is there really a deep inherent tradeoff between risk of entrenchment and stability of income, or is there some way we could get both? responses to negative contributions this round also introduced negative contributions, a feature proposed in my review of the previous round. but as with recurring contributions, very few people made negative contributions, to the point where their impact on the results was negligible. also, there was active opposition to negative contributions: source: honestly i have no idea, someone else sent it to me and they forgot where they found it. sorry :( the main source of opposition was basically what i predicted in the previous round. adding a mechanism that allows people to penalize others, even if deservedly so, can have tricky and easily harmful social consequences. some people even opposed the negative contribution mechanism to the point where they took care to give positive contributions to everyone who received a negative contribution. how do we respond? to me it seems clear that, in the long run, some mechanism of filtering out bad projects, and ideally compensating for overexcitement into good projects, will have to exist. it doesn't necessarily need to be integrated as a symmetric part of the qf, but there does need to be a filter of some form. and this mechanism, whatever form it will take, invariably opens up the possibility of the same social dynamics. so there is a challenge that will have to be solved no matter how we do it. one approach would be to hide more information: instead of just hiding who made a negative contribution, outright hide the fact that a negative contribution was made. many opponents of negative contributions explicitly indicated that they would be okay (or at least more okay) with such a model. and indeed (see the next section), this is a direction we will have to go anyway. but it would come at a cost effectively hiding negative contributions would mean not giving as much real-time feedback into what projects got how much funds. stepping up the fight against collusion this round saw much larger-scale attempts at collusion: it does seem clear that, at current scales, stronger protections against manipulation are goingto be required. the first thing that can be done is adding a stronger identity verification layer than github accounts; this is something that the gitcoin team is already working on. there is definitely a complex tradeoff between security and inclusiveness to be worked through, but it is not especially difficult to implement a first version. and if the identity problem is solved to a reasonable extent, that will likely be enough to prevent collusion at current scales. but in the longer term, we are going to need protection not just against manipulating the system by making many fake accounts, but also against collusion via bribes (explicit and implicit). maci is the solution that i proposed (and barry whitehat and co are implementing) to solve this problem. essentially, maci is a cryptographic construction that allows for contributions to projects to happen on-chain in a privacy-preserving, encrypted form, that allows anyone to cryptographically verify that the mechanism is being implemented correctly, but prevents participants from being able to prove to a third party that they made any particular contribution. unprovability means that if someone tries to bribe others to contribute to their project, the bribe recipients would have no way to prove that they actually contributed to that project, making the bribe unenforceable. benign "collusion" in the form of friends and family supporting each other would still happen, as people would not easily lie to each other at such small scales, but any broader collusion would be very difficult to maintain. however, we do need to think through some of the second-order consequences that integrating maci would introduce. the biggest blessing, and curse, of using maci is that contributions become hidden. identities necessarily become hidden, but even the exact timing of contributions would need to be hidden to prevent deanonymization through timing (to prove that you contributed, make the total amount jump up between 17:40 and 17:42 today). instead, for example, totals could be provided and updated once per day. note that as a corollary negative contributions would be hidden as well; they would only appear if they exceeded all positive contributions for an entire day (and if even that is not desired then the mechanism for when balances are updated could be tweaked to further hide downward changes). the challenge with hiding contributions is that we lose the "social proof" motivator for contributing: if contributions are unprovable you can't as easily publicly brag about a contribution you made. my best proposal for solving this is for the mechanism to publish one extra number: the total amount that a particular participant contributed (counting only projects that have received at least 10 contributors to prevent inflating one's number by self-dealing). individuals would then have a generic "proof-of-generosity" that they contributed some specific total amount, and could publicly state (without proof) what projects it was that they supported. but this is all a significant change to the user experience that will require multiple rounds of experimentation to get right. conclusions all in all, gitcoin grants is establishing itself as a significant pillar of the ethereum ecosystem that more and more projects are relying on for some or all of their support. while it has a relatively low amount of funding at present, and so inevitably underfunds almost everything it touches, we hope that over time we'll continue to see larger sources of funding for the matching pools appear. one option is mev auctions, another is that new or existing token projects looking to do airdrops could provide the tokens to a matching pool. a third is transaction fees of various applications. with larger amounts of funding, gitcoin grants could serve as a more significant funding stream though to get to that point, further iteration and work on fine-tuning the mechanism will be required. additionally, this round saw gitcoin grants' first foray into applications beyond ethereum with the health section. there is growing interest in quadratic funding from local government bodies and other non-blockchain groups, and it would be very valuable to see quadratic funding more broadly deployed in such contexts. that said, there are unique challenges there too. first, there's issues around onboarding people who do not already have cryptocurrency. second, the ethereum community is naturally expert in the needs of the ethereum community, but neither it nor average people are expert in, eg. medical support for the coronavirus pandemic. we should expect quadratic funding to perform worse when the participants are not experts in the domain they're being asked to contribute to. will non-blockchain uses of qf focus on domains where there's a clear local community that's expert in its own needs, or will people try larger-scale deployments soon? if we do see larger-scale deployments, how will those turn out? there's still a lot of questions to be answered. dark mode toggle what kind of layer 3s make sense? 2022 sep 17 see all posts special thanks to georgios konstantopoulos, karl floersch and the starkware team for feedback and review. one topic that often re-emerges in layer-2 scaling discussions is the concept of "layer 3s". if we can build a layer 2 protocol that anchors into layer 1 for security and adds scalability on top, then surely we can scale even more by building a layer 3 protocol that anchors into layer 2 for security and adds even more scalability on top of that? a simple version of this idea goes: if you have a scheme that can give you quadratic scaling, can you stack the scheme on top of itself and get exponential scaling? ideas like this include my 2015 scalability paper, the multi-layer scaling ideas in the plasma paper, and many more. unfortunately, such simple conceptions of layer 3s rarely quite work out that easily. there's always something in the design that's just not stackable, and can only give you a scalability boost once limits to data availability, reliance on l1 bandwidth for emergency withdrawals, or many other issues. newer ideas around layer 3s, such as the framework proposed by starkware, are more sophisticated: they aren't just stacking the same thing on top of itself, they're assigning the second layer and the third layer different purposes. some form of this approach may well be a good idea if it's done in the right way. this post will get into some of the details of what might and might not make sense to do in a triple-layered architecture. why you can't just keep scaling by stacking rollups on top of rollups rollups (see my longer article on them here) are a scaling technology that combines different techniques to address the two main scaling bottlenecks of running a blockchain: computation and data. computation is addressed by either fraud proofs or snarks, which rely on a very small number of actors to process and verify each block, requiring everyone else to perform only a tiny amount of computation to check that the proving process was done correctly. these schemes, especially snarks, can scale almost without limit; you really can just keep making "a snark of many snarks" to scale even more computation down to a single proof. data is different. rollups use a collection of compression tricks to reduce the amount of data that a transaction needs to store on-chain: a simple currency transfer decreases from ~100 to ~16 bytes, an erc20 transfer in an evm-compatible chain from ~180 to ~23 bytes, and a privacy-preserving zk-snark transaction could be compressed from ~600 to ~80 bytes. about 8x compression in all cases. but rollups still need to make data available on-chain in a medium that users are guaranteed to be able to access and verify, so that users can independently compute the state of the rollup and join as provers if existing provers go offline. data can be compressed once, but it cannot be compressed again if it can, then there's generally a way to put the logic of the second compressor into the first, and get the same benefit by compressing once. hence, "rollups on top of rollups" are not something that can actually provide large gains in scalability though, as we will see below, such a pattern can serve other purposes. so what's the "sane" version of layer 3s? well, let's look at what starkware, in their post on layer 3s, advocates. starkware is made up of very smart cryptographers who are actually sane, and so if they are advocating for layer 3s, their version will be much more sophisticated than "if rollups compress data 8x, then obviously rollups on top of rollups will compress data 64x". here's a diagram from starkware's post: a few quotes: an example of such an ecosystem is depicted in diagram 1. its l3s include: a starknet with validium data availability, e.g., for general use by applications with extreme sensitivity to pricing. app-specific starknet systems customized for better application performance, e.g., by employing designated storage structures or data availability compression. starkex systems (such as those serving dydx, sorare, immutable, and deversifi) with validium or rollup data availability, immediately bringing battle-tested scalability benefits to starknet. privacy starknet instances (in this example also as an l4) to allow privacy-preserving transactions without including them in public starknets. we can compress the article down into three visions of what "l3s" are for: l2 is for scaling, l3 is for customized functionality, for example privacy. in this vision there is no attempt to provide "scalability squared"; rather, there is one layer of the stack that helps applications scale, and then separate layers for customized functionality needs of different use cases. l2 is for general-purpose scaling, l3 is for customized scaling. customized scaling might come in different forms: specialized applications that use something other than the evm to do their computation, rollups whose data compression is optimized around data formats for specific applications (including separating "data" from "proofs" and replacing proofs with a single snark per block entirely), etc. l2 is for trustless scaling (rollups), l3 is for weakly-trusted scaling (validiums). validiums are systems that use snarks to verify computation, but leave data availability up to a trusted third party or committee. validiums are in my view highly underrated: in particular, many "enterprise blockchain" applications may well actually be best served by a centralized server that runs a validium prover and regularly commits hashes to chain. validiums have a lower grade of security than rollups, but can be vastly cheaper. all three of these visions are, in my view, fundamentally reasonable. the idea that specialized data compression requires its own platform is probably the weakest of the claims it's quite easy to design a layer 2 with a general-purpose base-layer compression scheme that users can automatically extend with application-specific sub-compressors but otherwise the use cases are all sound. but this still leaves open one large question: is a three-layer structure the right way to accomplish these goals? what's the point of validiums, and privacy systems, and customized environments, anchoring into layer 2 instead of just anchoring into layer 1? the answer to this question turns out to be quite complicated. which one is actually better? does depositing and withdrawing become cheaper and easier within a layer 2's sub-tree? one possible argument for the three-layer model over the two-layer model is: a three-layer model allows an entire sub-ecosystem to exist within a single rollup, which allows cross-domain operations within that ecosystem to happen very cheaply, without needing to go through the expensive layer 1. but as it turns out, you can do deposits and withdrawals cheaply even between two layer 2s (or even layer 3s) that commit to the same layer 1! the key realization is that tokens and other assets do not have to be issued in the root chain. that is, you can have an erc20 token on arbitrum, create a wrapper of it on optimism, and move back and forth between the two without any l1 transactions! let us examine how such a system works. there are two smart contracts: the base contract on arbitrum, and the wrapper token contract on optimism. to move from arbitrum to optimism, you would send your tokens to the base contract, which would generate a receipt. once arbitrum finalizes, you can take a merkle proof of that receipt, rooted in l1 state, and send it into the wrapper token contract on optimism, which verifies it and issues you a wrapper token. to move tokens back, you do the same thing in reverse. even though the merkle path needed to prove the deposit on arbitrum goes through the l1 state, optimism only needs to read the l1 state root to process the deposit no l1 transactions required. note that because data on rollups is the scarcest resource, a practical implementation of such a scheme would use a snark or a kzg proof, rather than a merkle proof directly, to save space. such a scheme has one key weakness compared to tokens rooted on l1, at least on optimistic rollups: depositing also requires waiting the fraud proof window. if a token is rooted on l1, withdrawing from arbitrum or optimism back to l1 takes a week delay, but depositing is instant. in this scheme, however, both depositing and withdrawing take a week delay. that said, it's not clear that a three-layer architecture on optimistic rollups is better: there's a lot of technical complexity in ensuring that a fraud proof game happening inside a system that itself runs on a fraud proof game is safe. fortunately, neither of these issues will be a problem on zk rollups. zk rollups do not require a week-long waiting window for security reasons, but they do still require a shorter window (perhaps 12 hours with first-generation technology) for two other reasons. first, particularly the more complex general-purpose zk-evm rollups need a longer amount of time to cover non-parallelizable compute time of proving a block. second, there is the economic consideration of needing to submit proofs rarely to minimize the fixed costs associated with proof transactions. next-gen zk-evm technology, including specialized hardware, will solve the first problem, and better-architected batch verification can solve the second problem. and it's precisely the issue of optimizing and batching proof submission that we will get into next. rollups and validiums have a confirmation time vs fixed cost tradeoff. layer 3s can help fix this. but what else can? the cost of a rollup per transaction is cheap: it's just 16-60 bytes of data, depending on the application. but rollups also have to pay a high fixed cost every time they submit a batch of transactions to chain: 21000 l1 gas per batch for optimistic rollups, and more than 400,000 gas for zk rollups (millions of gas if you want something quantum-safe that only uses starks). of course, rollups could simply choose to wait until there's 10 million gas worth of l2 transactions to submit a batch, but this would give them very long batch intervals, forcing users to wait much longer until they get a high-security confirmation. hence, they have a tradeoff: long batch intervals and optimum costs, or shorter batch intervals and greatly increased costs. to give us some concrete numbers, let us consider a zk rollup that has 600,000 gas per-batch costs and processes fully optimized erc20 transfers (23 bytes), which cost 368 gas per transaction. suppose that this rollup is in early to mid stages of adoption, and is averaging 5 tps. we can compute gas per transaction vs batch intervals: batch interval gas per tx (= tx cost + batch cost / (tps * batch interval)) 12s (one per ethereum block) 10368 1 min 2368 10 min 568 1 h 401 if we're entering a world with lots of customized validiums and application-specific environments, then many of them will do much less than 5 tps. hence, tradeoffs between confirmation time and cost start to become a very big deal. and indeed, the "layer 3" paradigm does solve this! a zk rollup inside a zk rollup, even implemented naively, would have fixed costs of only ~8,000 layer-1 gas (500 bytes for the proof). this changes the table above to: batch interval gas per tx (= tx cost + batch cost / (tps * batch interval)) 12s (one per ethereum block) 501 1 min 394 10 min 370 1 h 368 problem basically solved. so are layer 3s good? maybe. but it's worth noticing that there is a different approach to solving this problem, inspired by erc 4337 aggregate verification. the strategy is as follows. today, each zk rollup or validium accepts a state root if it receives a proof proving that \(s_{new} = stf(s_{old}, d)\): the new state root must be the result of correctly processing the transaction data or state deltas on top of the old state root. in this new scheme, the zk rollup would accept a message from a batch verifier contract that says that it has verified a proof of a batch of statements, where each of those statements is of the form \(s_{new} = stf(s_{old}, d)\). this batch proof could be constructed via a recursive snark scheme or halo aggregation. this would be an open protocol: any zk-rollup could join, and any batch prover could aggregate proofs from any compatible zk-rollup, and would get compensated by the aggregator with a transaction fee. the batch handler contract would verify the proof once, and then pass off a message to each rollup with the \((s_{old}, s_{new}, d)\) triple for that rollup; the fact that the triple came from the batch handler contract would be evidence that the transition is valid. the cost per rollup in this scheme could be close to 8000 if it's well-optimized: 5000 for a state write adding the new update, 1280 for the old and new root, and an extra 1720 for miscellaneous data juggling. hence, it would give us the same savings. starkware actually has something like this already, called sharp, though it is not (yet) a permissionless open protocol. one response to this style of approach might be: but isn't this actually just another layer 3 scheme? instead of base layer <rollup <validium, you have base layer <batch mechanism <rollup or validium. from some philosophical architectural standpoint, this may be true. but there is an important difference: instead of the middle layer being a complicated full evm system, the middle layer is a simplified and highly specialized object, and so it is more likely to be secure, it is more likely to be built at all without needing yet another specialized token, and it is more likely to be governance-minimized and not change over time. conclusion: what even is a "layer"? a three-layer scaling architecture that consists of stacking the same scaling scheme on top of itself generally does not work well. rollups on top of rollups, where the two layers of rollups use the same technology, certainly do not. a three-layer architecture where the second layer and third layer have different purposes, however, can work. validiums on top of rollups do make sense, even if they're not certain to be the long-term best way of doing things. once we start getting into the details of what kind of architecture makes sense, however, we get into the philosophical question: what is a "layer" and what is not? the base layer <batch mechanism <rollup or validium pattern does the same job as a base layer <rollup <rollup or validium pattern. but in terms of how it works, a proof aggregation layer looks more like erc-4337 than like a rollup. typically, we don't refer to erc-4337 as a "layer 2". similarly, we don't refer to tornado cash as a "layer 2" and so if we were to be consistent, we would not refer to a privacy-focused sub-system that lives on top of a layer 2 as a layer 3. so there is an unresolved semantics debate of what deserves the title of "layer" in the first place. there are many possible schools of thought on this. my personal preference would be to keep the term "layer 2" restricted to things with the following properties: their purpose is to increase scalability they follow the "blockchain within a blockchain" pattern: they have their own mechanism for processing transactions and their own internal state they inherit the full security of the ethereum chain so, optimistic rollups and zk rollups are layer 2s, but validiums, proof aggregation schemes, erc 4337, on-chain privacy systems and solidity are something else. it may make sense to call some of them layer 3s, but probably not all of them; in any case, it seems premature to settle definitions while the architecture of the multi-rollup ecosystem is far from set in stone and most of the discussion is happening only in theory. that said, the language debate is less important than the technical question of which constructions actually make the most sense. there is clearly an important role to be played by "layers" of some kind that serve non-scaling needs like privacy, and there is clearly is an important function of proof aggregation that needs to be filled somehow, and preferably by an open protocol. but at the same time, there are good technical reasons to make the intermediary layers that connect user-facing environments to the layer 1 as simple as possible; the "glue layer" being an evm rollup is probably not the right approach in many cases. i suspect more sophisticated (and simpler) constructions such as those described in this post will start to have a bigger role to play as the layer 2 scaling ecosystem matures. security alert 1 [windows+alethzero] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security alert 1 [windows+alethzero] posted by jutta steiner on august 7, 2015 security this affects users of alethzero gui client on windows. users of eth cli client or not on the windows platform are unlikely to be affected but should take action detailed below. users of frontier command line interface geth are unaffected. issue description: while setting privacy permissions on the keys directory, insufficient error handling can cause the key files to not be written; this may be widespread on the windows platform. as such, current versions of alethzero and eth may include identities for which there exists no underlying key. ether presale claim functionality of alethzero may result in funds automatically being transferred to these lost identities. workaround: users of alethzero version 0.9.39 and earlier should not use the “claim presale wallet” function; users of alethzero and eth versions 0.9.39 and earlier should not attempt to mine or receive funds into their addresses. users of eth and alethzero on all platforms should consider themselves safe once they have confirmed that they do indeed have the underlying key. to check (with your existing setup) run: ethkey.exe --list you may assume that all listed addresses do indeed have a key behind them and are not suffering from this issue. remedial action taken by ethereum: new hotfix released with changes: identities for which there are no underlying keys are no longer displayed. key files are written regardless of whether setting directory permissions succeeded. fix: versions 0.9.40 and onwards, available from circa 2015.08.07 18:30 cest. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle against overuse of the gini coefficient 2021 jul 29 see all posts special thanks to barnabe monnot and tina zhen for feedback and review the gini coefficient (also called the gini index) is by far the most popular and widely known measure of inequality, typically used to measure inequality of income or wealth in some country, territory or other community. it's popular because it's easy to understand, with a mathematical definition that can easily be visualized on a graph. however, as one might expect from any scheme that tried to reduce inequality to a single number, the gini coefficient also has its limits. this is true even in its original context of measuring income and wealth inequality in countries, but it becomes even more true when the gini coefficient is transplanted into other contexts (particularly: cryptocurrency). in this post i will talk about some of the limits of the gini coefficient, and propose some alternatives. what is the gini coefficient? the gini coefficient is a measure of inequality introduced by corrado gini in 1912. it is typically used to measure inequality of income and wealth of countries, though it is also increasingly being used in other contexts. there are two equivalent definitions of the gini coefficient: area-above-curve definition: draw the graph of a function, where \(f(p)\) equals the share of total income earned by the lowest-earning portion of the population (eg. \(f(0.1)\) is the share of total income earned by the lowest-earning 10%). the gini coefficient is the area between that curve and the \(y=x\) line, as a portion of the whole triangle: average-difference definition: the gini coefficient is half the average difference of incomes between each all possible pairs of individuals, divided by the mean income. for example, in the above example chart, the four incomes are [1, 2, 4, 8], so the 16 possible differences are [0, 1, 3, 7, 1, 0, 2, 6, 3, 2, 0, 4, 7, 6, 4, 0]. hence the average difference is 2.875 and the mean income is 3.75, so gini = \(\frac{2.875}{2 * 3.75} \approx 0.3833\). it turns out that the two are mathematically equivalent (proving this is an exercise to the reader)! what's wrong with the gini coefficient? the gini coefficient is attractive because it's a reasonably simple and easy-to-understand statistic. it might not look simple, but trust me, pretty much everything in statistics that deals with populations of arbitrary size is that bad, and often much worse. here, stare at the formula of something as basic as the standard deviation: \(\sigma = \frac{\sum_{i=1}^n x_i^2}{n} (\frac{\sum_{i=1}^n x_i}{n})^2\) and here's the gini: \(g = \frac{2 * \sum_{i=1}^n i*x_i}{n * \sum_{i=1}^n x_i} \frac{n+1}{n}\) it's actually quite tame, i promise! so, what's wrong with it? well, there are lots of things wrong with it, and people have written lots of articles about various problems with the gini coefficient. in this article, i will focus on one specific problem that i think is under-discussed about the gini as a whole, but that has particular relevance to analyzing inequality in internet communities such as blockchains. the gini coefficient combines together into a single inequality index two problems that actually look quite different: suffering due to lack of resources and concentration of power. to understand the difference between the two problems more clearly, let's look at two dystopias: dystopia a: half the population equally shares all the resources, everyone else has none dystopia b: one person has half of all the resources, everyone else equally shares the remaining half here are the lorenz curves (fancy charts like we saw above) for both dystopias: clearly, neither of those two dystopias are good places to live. but they are not-very-nice places to live in very different ways. dystopia a gives each resident a coin flip between unthinkably horrific mass starvation if they end up on the left half on the distribution and egalitarian harmony if they end up on the right half. if you're thanos, you might actually like it! if you're not, it's worth avoiding with the strongest force. dystopia b, on the other hand, is brave new world-like: everyone has decently good lives (at least at the time when that snapshot of everyone's resources is taken), but at the high cost of an extremely undemocratic power structure where you'd better hope you have a good overlord. if you're curtis yarvin, you might actually like it! if you're not, it's very much worth avoiding too. these two problems are different enough that they're worth analyzing and measuring separately. and this difference is not just theoretical. here is a chart showing share of total income earned by the bottom 20% (a decent proxy for avoiding dystopia a) versus share of total income earned by the top 1% (a decent proxy for being near dystopia b): sources: https://data.worldbank.org/indicator/si.dst.frst.20 (merging 2015 and 2016 data) and http://hdr.undp.org/en/indicators/186106. the two are clearly correlated (coefficient -0.62), but very far from perfectly correlated (the high priests of statistics apparently consider 0.7 to be the lower threshold for being "highly correlated", and we're even under that). there's an interesting second dimension to the chart that can be analyzed what's the difference between a country where the top 1% earn 20% of the total income and the bottom 20% earn 3% and a country where the top 1% earn 20% and the bottom 20% earn 7%? alas, such an exploration is best left to other enterprising data and culture explorers with more experience than myself. why gini is very problematic in non-geographic communities (eg. internet/crypto communities) wealth concentration within the blockchain space in particular is an important problem, and it's a problem worth measuring and understanding. it's important for the blockchain space as a whole, as many people (and us senate hearings) are trying to figure out to what extent crypto is truly anti-elitist and to what extent it's just replacing old elites with new ones. it's also important when comparing different cryptocurrencies with each other. share of coins explicitly allocated to specific insiders in a cryptocurrency's initial supply is one type of inequality. note that the ethereum data is slightly wrong: the insider and foundation shares should be 12.3% and 4.2%, not 15% and 5%. given the level of concern about these issues, it should be not at all surprising that many people have tried computing gini indices of cryptocurrencies: the observed gini index for staked eos tokens (2018) gini coefficients of cryptocurrencies (2018) measuring decentralization in bitcoin and ethereum using multiple metrics and granularities (2021, includes gini and 2 other metrics) nouriel roubini comparing bitcoin's gini to north korea (2018) on-chain insights in the cryptocurrency markets (2021, uses gini to measure concentration) and even earlier than that, we had to deal with this sensationalist article from 2014: in addition to common plain methodological mistakes (often either mixing up income vs wealth inequality, mixing up users vs accounts, or both) that such analyses make quite frequently, there is a deep and subtle problem with using the gini coefficient to make these kinds of comparisons. the problem lies in key distinction between typical geographic communities (eg. cities, countries) and typical internet communities (eg. blockchains): a typical resident of a geographic community spends most of their time and resources in that community, and so measured inequality in a geographic community reflects inequality in total resources available to people. but in an internet community, measured inequality can come from two sources: (i) inequality in total resources available to different participants, and (ii) inequality in level of interest in participating in the community. the average person with $15 in fiat currency is poor and is missing out on the ability to have a good life. the average person with $15 in cryptocurrency is a dabbler who opened up a wallet once for fun. inequality in level of interest is a healthy thing; every community has its dabblers and its full-time hardcore fans with no life. so if a cryptocurrency has a very high gini coefficient, but it turns out that much of this inequality comes from inequality in level of interest, then the number points to a much less scary reality than the headlines imply. cryptocurrencies, even those that turn out to be highly plutocratic, will not turn any part of the world into anything close to dystopia a. but badly-distributed cryptocurrencies may well look like dystopia b, a problem compounded if coin voting governance is used to make protocol decisions. hence, to detect the problems that cryptocurrency communities worry about most, we want a metric that captures proximity to dystopia b more specifically. an alternative: measuring dystopia a problems and dystopia b problems separately an alternative approach to measuring inequality involves directly estimating suffering from resources being unequally distributed (that is, "dystopia a" problems). first, start with some utility function representing the value of having a certain amount of money. \(log(x)\) is popular, because it captures the intuitively appealing approximation that doubling one's income is about as useful at any level: going from $10,000 to $20,000 adds the same utility as going from $5,000 to $10,000 or from $40,000 to $80,000). the score is then a matter of measuring how much utility is lost compared to if everyone just got the average income: \(log(\frac{\sum_{i=1}^n x_i}{n}) \frac{\sum_{i=1}^n log(x_i)}{n}\) the first term (log-of-average) is the utility that everyone would have if money were perfectly redistributed, so everyone earned the average income. the second term (average-of-log) is the average utility in that economy today. the difference represents lost utility from inequality, if you look narrowly at resources as something used for personal consumption. there are other ways to define this formula, but they end up being close to equivalent (eg. the 1969 paper by anthony atkinson suggested an "equally distributed equivalent level of income" metric which, in the \(u(x) = log(x)\) case, is just a monotonic function of the above, and the theil l index is perfectly mathematically equivalent to the above formula). to measure concentration (or "dystopia b" problems), the herfindahl-hirschman index is an excellent place to start, and is already used to measure economic concentration in industries: \(\frac{\sum_{i=1}^n x_i^2}{(\sum_{i=1}^n x_i)^2}\) or for you visual learners out there: herfindahl-hirschman index: green area divided by total area. there are other alternatives to this; the theil t index has some similar properties though also some differences. a simpler-and-dumber alternative is the nakamoto coefficient: the minimum number of participants needed to add up to more than 50% of the total. note that all three of these concentration indices focus heavily on what happens near the top (and deliberately so): a large number of dabblers with a small quantity of resources contributes little or nothing to the index, while the act of two top participants merging can make a very big change to the index. for cryptocurrency communities, where concentration of resources is one of the biggest risks to the system but where someone only having 0.00013 coins is not any kind of evidence that they're actually starving, adopting indices like this is the obvious approach. but even for countries, it's probably worth talking about, and measuring, concentration of power and suffering from lack of resources more separately. that said, at some point we have to move beyond even these indices. the harms from concentration are not just a function of the size of the actors; they are also heavily dependent on the relationships between the actors and their ability to collude with each other. similarly, resource allocation is network-dependent: lack of formal resources may not be that harmful if the person lacking resources has an informal network to tap into. but dealing with these issues is a much harder challenge, and so we do also need the simpler tools while we still have less data to work with. chain reorganisation depth expectations | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search chain reorganisation depth expectations posted by gavin wood on august 8, 2015 research & development this following shows our current and planned expectations concerning maximum likely chain-reorganisation depth. we would not consider transactions within this depth to have an exceptionally high chance of being permanent. these are our own expectations only and do not constitute any sort of guarantee. they are derived from theoretical considerations, ongoing empirical data, human factors in contingency planning and the past experience of our security team. as with all things in the peer-to-peer space the risk is entirely with the individual operator. in much the same way as many in the space, we will be monitoring the chain for any signs of protocol-level issues. if we have any reason to suspect that there is a protocol level issue we will update these expectations accordingly; the updates will be posted in the forums and on the official blog. all those who are interested in our expectations and recommendations would do well to keep themselves abreast of the blog. roadmap until 2015/08/08 18:00:00 cest: 6000 from 2015/08/08 18:00:00 cest, 3000 (approx 12 hours) (1 day) from 2015/08/09 18:00:00 cest, 1500 (approx 6 hours) (3 days) from 2015/08/12 18:00:00 cest, 750 (approx 3 hours) (3 days) from 2015/08/15 18:00:00 cest, 375 (approx 90 minutes) (rest of frontier) addendum 2015/08/08: you may be slightly perplexed as to the meaning of the "chain reorganisation depth". chain reorganisations happen when a node on the ethereum network (one which could belong to you, me, an exchange, a miner, whoever) realises that what it thought was the canonical chain turned out not to be. when this happens, the transactions in the latter part of its chain (i.e. the most recent transactions) are reverted and rather the transactions in the newer replacement are executed. with ethereum having a short target block time of 15s, this actually happens naturally rather often. because it takes time for the blocks to percolate through the network, it's easy for different parts of the network to have a different final block (or two, or perhaps even three) in normal operation since the miners often come up with them at roughly the same time. this is what we might call ephemeral forking. indeed, many of the ommers (né uncles) that you see in ethereum's network monitor were once assumed by some nodes to be the final block in canonical chain. when a re-organisation happens, or put another way, when the network reaches a more global consensus that it had earlier and a fork is resolved, the nodes that had the now out-dated chain "reorganise" their chain, throwing away the recent and no-longer canonical blocks. transactions are reverted and others executed to get in line with the other path of the fork. transactions can be mutually exclusive, like cheques; if i have 100inmyaccountandiwritetwochequeseachfor100 in my account and i write two cheques each for 100inmyaccountandiwritetwochequeseachfor100, the order is very important since they can't both be paid. this means that a reorganisation could result in the reversion of one transaction and the execution of another, mutually exclusive transaction. as such if you're going to do an irreversible action on the back of a transaction being in the chain, it's very important to know the risks regarding reorganisation. roughly speaking, the chances of a reorganisation occurring reduce substantially the farther from the end you get. that is, the chance of a reorganisation happening that alters the final three blocks is much less than the chance of one that alters the final block alone. this is because the consensus algorithm is constantly striving to end up at a common agreement over what the chain is. as long as there isn't consensus (and thus potential for a reorganisation), it's not in a stable state and will sooner or later topple into agreement. we call the number of blocks affected by the reorganisation the depth of the reorganisation. in general reorganisations happen automatically and safely, however, anyone making real-world decisions based upon transactions on the chain needs to be aware of reorganisations happening and, most importantly, must make a judgement decision on how deep a transaction must get in the apparent chain before they decide it is the final chain and not merely a temporary fork than will eventually be reverted and resolved. the decision of how deep to wait is, in bitcoin terms, called the number of confirmations. our (somewhat large) expectations of possible reorganisation depth (which may very well inform confirmation numbers) come from the fact that the protocol is immature, that human factors are involved in any remedial action and that substantial amounts could be at stake. basically, it's the frontier. there are scenarios, especially those involving adversaries ("51%" attackers) that we have devised in which we believe fairly large numbers are indeed warranted at this initial stage. ultimately, of course, we can only advise and inform: the risk on how many "confirmations" to wait (or not) as with that of all operational decisions, lies with you. welcome to freedom :-) previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements understanding serenity, part i: abstraction | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search understanding serenity, part i: abstraction posted by vitalik buterin on december 24, 2015 research & development special thanks to gavin wood for prompting my interest into abstraction improvements, and martin becze, vlad zamfir and dominic williams for ongoing discussions. for a long time we have been public about our plans to continue improving the ethereum protocol over time and our long development roadmap, learning from our mistakes that we either did not have the opportunity to fix in time for 1.0 or only realized after the fact. however, the ethereum protocol development cycle has started up once again, with a homestead release coming very soon, and us quietly starting to develop proof-of-concepts for the largest milestone that we had placed for ourselves in our development roadmap: serenity. serenity is intended to have two major feature sets: abstraction, a concept that i initially expanded on in this blog post here, and casper, our security-deposit-based proof of stake algorithm. additionally, we are exploring the idea of adding at least the scaffolding that will allow for the smooth deployment over time of our scalability proposals, and at the same time completely resolve parallelizability concerns brought up here an instant very large gain for private blockchain instances of ethereum with nodes being run in massively multi-core dedicated servers, and even the public chain may see a 2-5x improvement in scalability. over the past few months, research on casper and formalization of scalability and abstraction (eg. with eip 101) have been progressing at a rapid pace between myself, vlad zamfir, lucius greg meredith and a few others, and now i am happy to announce that the first proof of concept release for serenity, albeit in a very limited form suitable only for testing, is now available. the poc can be run by going into the ethereum directory and running python test.py (make sure to download and install the latest serpent from https://github.com/ethereum/serpent, develop branch); if the output looks something like this then you are fine: vub@vub-thinkpad-x250 15:01:03 serenity/ethereum: python test.py reverting 940534 gas from account 0x0000000000000000000000000000000000000000 to account 0x98c78be58d729dcdc3de9efb3428820990e4e3bf with data 0x warning (file "casper.se.py", line 74, char 0): warning: function return type inconsistent! running with 13 maximum nodes warning (file "casper.se.py", line 74, char 0): warning: function return type inconsistent! warning (file "casper.se.py", line 74, char 0): warning: function return type inconsistent! length of validation code: 57 length of account code: 0 joined with index 0 length of validation code: 57 length of account code: 0 joined with index 1 length of validation code: 57 this is a simulation of 13 nodes running the casper+serenity protocol at a 5-second block time; this is fairly close to the upper limit of what the client can handle at the moment, though note that (i) this is python, and c++ and go will likely show much higher performance, and (ii) this is all nodes running on one computer at the same time, so in a more "normal" environment it means you can expect python casper to be able to handle at least ~169 nodes (though, on the other hand, we want consensus overhead to be much less than 100% of cpu time, so these two caveats combined do not mean that you should expect to see casper running with thousands of nodes!). if your computer is too slow to handle the 13 nodes, try python test.py 10 to run the simulation with 10 nodes instead (or python test.py 7 for 7 nodes, etc). of course, research on improving casper's efficiency, though likely at the cost of somewhat slower convergence to finality, is still continuing, and these problems should reduce over time. the network.py file simulates a basic p2p network interface; future work will involve swapping this out for actual computers running on a real network. the code is split up into several main files as follows: serenity_blocks.py the code that describes the block class, the state class and the block and transaction-level transition functions (about 2x simpler than before) serenity_transactions.py the code that describes transactions (about 2x simpler than before) casper.se.py the serpent code for the casper contract, which incentivizes correct betting bet.py casper betting strategy and full client implementation ecdsa_accounts.py account code that allows you to replicate the account validation functionality available today in a serenity context test.py the testing script config.py config parameters vm.py the virtual machine (faster implementation at fastvm.py) network.py the network simulator for this article, we will focus on the abstraction features and so serenity_blocks.py, ecdsa_accounts.py and serenity_transactions.py are most critical; for the next article discussing casper in serenity, casper.se.py and bet.py will be a primary focus. abstraction and accounts currently, there are two types of accounts in ethereum: externally owned accounts, controlled by a private key, and contracts, controlled by code. for externally owned accounts, we specify a particular digital signature algorithm (secp256k1 ecdsa) and a particular sequence number (aka. nonce) scheme, where every transaction must include a sequence number one higher than the previous, in order to prevent replay attacks. the primary change that we will make in order to increase abstraction is this: rather than having these two distinct types of accounts, we will now have only one contracts. there is also a special "entry point" account, 0x0000000000000000000000000000000000000000, that anyone can send from by sending a transaction. hence, instead of the signature+nonce verification logic of accounts being in the protocol, it is now up to the user to put this into a contract that will be securing their own account. the simplest kind of contract that is useful is probably the ecdsa verification contract, which simply provides the exact same functionality that is available right now: transactions pass through only if they have valid signatures and sequence numbers, and the sequence number is incremented by 1 if a transaction succeeds. the code for the contract looks as follows: # we assume that data takes the following schema: # bytes 0-31: v (ecdsa sig) # bytes 32-63: r (ecdsa sig) # bytes 64-95: s (ecdsa sig) # bytes 96-127: sequence number (formerly called "nonce") # bytes 128-159: gasprice # bytes 172-191: to # bytes 192-223: value # bytes 224+: data # get the hash for transaction signing ~mstore(0, ~txexecgas()) ~calldatacopy(32, 96, ~calldatasize() 96) ~mstore(0, ~sha3(0, ~calldatasize() 64)) ~calldatacopy(32, 0, 96) # call ecrecover contract to get the sender ~call(5000, 1, 0, 0, 128, 0, 32) # check sender correctness; exception if not if ~mload(0) != 0x82a978b3f5962a5b0957d9ee9eef472ee55b42f1: ~invalid() # sequence number operations with minusone = ~sub(0, 1): with curseq = self.storage[minusone]: # check sequence number correctness, exception if not if ~calldataload(96) != curseq: ~invalid() # increment sequence number self.storage[minusone] = curseq + 1 # make the sub-call and discard output with x = ~msize(): ~call(msg.gas 50000, ~calldataload(160), ~calldataload(192), 160, ~calldatasize() 224, x, 1000) # pay for gas ~mstore(0, ~calldataload(128)) ~mstore(32, (~txexecgas() msg.gas + 50000)) ~call(12000, ether, 0, 0, 64, 0, 0) ~return(x, ~msize() x) this code would sit as the contract code of the user's account; if the user wants to send a transaction, they would send a transaction (from the zero address) to this account, encoding the ecdsa signature, the sequence number, the gasprice, destination address, ether value and the actual transaction data using the encoding specified above in the code. the code checks the signature against the transaction gas limit and the data provided, and then checks the sequence number, and if both are correct it then increments the sequence number, sends the desired message, and then at the end sends a second message to pay for gas (note that miners can statically analyze accounts and refuse to process transactions sending to accounts that do not have gas payment code at the end). an important consequence of this is that serenity introduces a model where all transactions (that satisfy basic formatting checks) are valid; transactions that are currently "invalid" will in serenity simply have no effect (the invalid opcode in the code above simply points to an unused opcode, immediately triggering an exit from code execution). this does mean that transaction inclusion in a block is no longer a guarantee that the transaction was actually executed; to substitute for this, every transaction now gets a receipt entry that specifies whether or not it was successfully executed, providing one of three return codes: 0 (transaction not executed due to block gas limit), 1 (transaction executed but led to error), 2 (transaction executed successfully); more detailed information can be provided if the transaction returns data (which is now auto-logged) or creates its own logs. the main very large benefit of this is that it gives users much more freedom to innovate in the area of account policy; possible directions include: bitcoin-style multisig, where an account expects signatures from multiple public keys at the same time before sending a transaction, rather than accepting signatures one at a time and saving intermediate results in storage other elliptic curves, including ed25519 better integration for more advanced crypto, eg. ring signatures, threshold signatures, zkps more advanced sequence number schemes that allow for higher degrees of parallelization, so that users can send many transactions from one account and have them included more quickly; think a combination of a traditional sequence number and a bitmask. one can also include timestamps or block hashes into the validity check in various clever ways. utxo-based token management some people dislike the fact that ethereum uses accounts instead of bitcoin's "unspent transaction output" (utxo) model for managing token ownership, in part for privacy reasons. now, you can create a system inside ethereum that actually is utxo-based, and serenity no longer explicitly "privileges" one over the other. innovation in payment schemes for some dapps, "contract pays" is a better model than "sender pays" as senders may not have any ether; now, individual dapps can implement such models, and if they are written in a way that miners can statically analyze and determine that they actually will get paid, then they can immediately accept them (essentially, this provides what rootstock is trying to do with optional author-pays, but in a much more abstract and flexible way). stronger integration for "ethereum alarm clock"-style applications the verification code for an account doesn't have to check for signatures, it could also check for merkle proofs of receipts, state of other accounts, etc in all of these cases, the primary point is that through abstraction all of these other mechanisms become much easier to code as there is no longer a need to create a "pass-through layer" to feed the information in through ethereum's default signature scheme; when no application is special, every application is. one particular interesting consequence is that with the current plan for serenity, ethereum will be optionally quantum-safe; if you are scared of the nsa having access to a quantum computer, and want to protect your account more securely, you can personally switch to lamport signatures at any time. proof of stake further bolsters this, as even if the nsa had a quantum computer and no one else they would not be able to exploit that to implement a 51% attack. the only cryptographic security assumption that will exist at protocol level in ethereum is collision-resistance of sha3. as a result of these changes, transactions are also going to become much simpler. instead of having nine fields, as is the case right now, transactions will only have four fields: destination address, data, start gas and init code. destination address, data and start gas are the same as they are now; "init code" is a field that can optionally contain contract creation code for the address that you are sending to. the reason for the latter mechanic is as follows. one important property that ethereum currently provides is the ability to send to an account before it exists; you do not need to already have ether in order to create a contract on the blockchain before you can receive ether. to allow this in serenity, an account's address can be determined from the desired initialization code for the account in advance, by using the formula sha3(creator + initcode) % 2**160 where creator is the account that created the contract (the zero account by default), and initcode is the initialization code for the contract (the output of running the initcode will become the contract code, just as is the case for creates right now). you can thus generate the initialization code for your contract locally, compute the address, and let others send to that address. then, once you want to send your first transaction, you include the init code in the transaction, and the init code will be executed automatically and the account created before proceeding to run the actual transaction (you can find this logic implemented here). abstraction and blocks another clean separation that will be implemented in serenity is the complete separation of blocks (which are now simply packages of transactions), state (ie. current contract storage, code and account balances) and the consensus layer. consensus incentivization is done inside a contract, and consensus-level objects (eg. pow, bets) should be included as transactions sent to a "consensus incentive manager contract" if one wishes to incentivize them. this should make it much easier to take the serenity codebase and swap out casper for any consensus algorithm tendermint, honeybadgerbft, subjective consensus or even plain old proof of work; we welcome research in this direction and aim for maximum flexibility. abstraction and storage currently, the "state" of the ethereum system is actually quite complex and includes many parts: balance, code, nonce and storage of accounts gas limit, difficulty, block number, timestamp the last 256 block hashes during block execution, the transaction index, receipt tree and the current gas used these data structures exist in various places, including the block state transition function, the state tree, the block header and previous block headers. in serenity, this will be simplified greatly: although many of these variables will still exist, they will all be moved to specialized contracts in storage; hence, the only concept of "state" that will continue to exist is a tree, which can mathematically be viewed as a mapping {address: {key: value} }. accounts will simply be trees; account code will be stored at key "" for each account (not mutable by sstore), balances will be stored in a specialized "ether contract" and sequence numbers will be left up to each account to determine how to store. receipts will also be moved to storage; they will be stored in a "log contract" where the contents get overwritten every block. this allows the state object in implementations to be simplified greatly; all that remains is a two-level map of tries. the scalability upgrade may increase this to three levels of tries (shard id, address, key) but this is not yet determined, and even then the complexity will be substantially smaller than today. note that the move of ether into a contract does not constitute total ether abstraction; in fact, it is arguably not that large a change from the status quo, as opcodes that deal with ether (the value parameter in call, balance, etc) still remain for backward-compatibility purposes. rather, this is simply a reorganization of how data is stored. future plans for poc2, the plan is to take abstraction even further. currently, substantial complexity still remains in the block and transaction-level state transition function (eg. updating receipts, gas limits, the transaction index, block number, stateroots); the goal will be to create an "entry point" object for transactions which handles all of this extra "boilerplate logic" that needs to be done per transaction, as well as a "block begins" and "block ends" entry point. a theoretical ultimate goal is to come up with a protocol where there is only one entry point, and the state transition function consists of simply sending a message from the zero address to the entry point containing the block contents as data. the objective here is to reduce the size of the actual consensus-critical client implementation as much as possible, pushing a maximum possible amount of logic directly into ethereum code itself; this ensures that ethereum's multi-client model can continue even with an aggressive development regime that is willing to accept hard forks and some degree of new complexity in order to achieve our goals of transaction speed and scalability without requiring an extremely large amount of ongoing development effort and security auditing. in the longer term, i intend to continue producing proof-of-concepts in python, while the casper team works together on improving the efficiency and proving the safety and correctness of the protocol; at some point, the protocol will be mature enough to handle a public testnet of some form, possibly (but not certainly) with real value on-chain in order to provide stronger incentives for people to try to "hack" casper they way that we inevitably expect that they will once the main chain goes live. this is only an initial step, although a very important one as it marks the first time when the research behind proof of stake and abstraction is finally moving from words, math on whiteboards and blog posts into a working implementation written in code. the next part of this series will discuss the other flagship feature of serenity, the casper consensus algorithm. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle prediction markets: tales from the election 2021 feb 18 see all posts special thanks to jeff coleman, karl floersch and robin hanson for critical feedback and review. trigger warning: i express some political opinions. prediction markets are a subject that has interested me for many years. the idea of allowing anyone in the public to make bets about future events, and using the odds at which these bets are made as a credibly neutral source of predicted probabilities of these events, is a fascinating application of mechanism design. closely related ideas, like futarchy, have always interested me as innovative tools that could improve governance and decision-making. and as augur and omen, and more recently polymarket, have shown, prediction markets are a fascinating application of blockchains (in all three cases, ethereum) as well. and the 2020 us presidential election, it seems like prediction markets are finally entering the limelight, with blockchain-based markets in particular growing from near-zero in 2016 to millions of dollars of volume in 2020. as someone who is closely interested in seeing ethereum applications cross the chasm into widespread adoption, this of course aroused my interest. at first, i was inclined to simply watch, and not participate myself: i am not an expert on us electoral politics, so why should i expect my opinion to be more correct than that of everyone else who was already trading? but in my twitter-sphere, i saw more and more arguments from very smart people whom i respected arguing that the markets were in fact being irrational and i should participate and bet against them if i can. eventually, i was convinced. i decided to make an experiment on the blockchain that i helped to create: i bought $2,000 worth of ntrump (tokens that pay $1 if trump loses) on augur. little did i know then that my position would eventually increase to $308,249, earning me a profit of over $56,803, and that i would make all of these remaining bets, against willing counterparties, after trump had already lost the election. what would transpire over the next two months would prove to be a fascinating case study in social psychology, expertise, arbitrage, and the limits of market efficiency, with important ramifications to anyone who is deeply interested in the possibilities of economic institution design. before the election my first bet on this election was actually not on a blockchain at all. when kanye announced his presidential bid in july, a political theorist whom i ordinarily quite respect for his high-quality and original thinking immediately claimed on twitter that he was confident that this would split the anti-trump vote and lead to a trump victory. i remember thinking at the time that this particular opinion of his was over-confident, perhaps even a result of over-internalizing the heuristic that if a viewpoint seems clever and contrarian then it is likely to be correct. so of course i offered to make a $200 bet, myself betting the boring conventional pro-biden view, and he honorably accepted. the election came up again on my radar in september, and this time it was the prediction markets that caught my attention. the markets gave trump a nearly 50% chance of winning, but i saw many very smart people in my twitter-sphere whom i respected pointing out that this number seemed far too high. this of course led to the familiar "efficient markets debate": if you can buy a token that gives you $1 if trump loses for $0.52, and trump's actual chance of losing is much higher, why wouldn't people just come in and buy the token until the price rises more? and if nobody has done this, who are you to think that you're smarter than everyone else? ne0liberal's twitter thread just before election day does an excellent job summarizing his case against prediction markets being accurate at that time. in short, the (non-blockchain) prediction markets that most people used at least prior to 2020 have all sorts of restrictions that make it difficult for people to participate with more than a small amount of cash. as a result, if a very smart individual or a professional organization saw a probability that they believed was wrong, they would only have a very limited ability to push the price in the direction that they believe to be correct. the most important restrictions that the paper points out are: low limits (well under $1,000) on how much each person can bet high fees (eg. a 5% withdrawal fee on predictit) and this is where i pushed back against ne0liberal in september: although the stodgy old-world centralized prediction markets may have low limits and high fees, the crypto markets do not! on augur or omen, there's no limit to how much someone can buy or sell if they think the price of some outcome token is too low or too high. and the blockchain-based prediction markets were following the same prices as predictit. if the markets really were over-estimating trump because high fees and low trading limits were preventing the more cool-headed traders from outbidding the overly optimistic ones, then why would blockchain-based markets, which don't have those issues, show the same prices? predictit augur the main response my twitter friends gave to this was that blockchain-based markets are highly niche, and very few people, particularly very few people who know much about politics, have easy access to cryptocurrency. that seemed plausible, but i was not too confident in that argument. and so i bet $2,000 against trump and went no further. the election then the election happened. after an initial scare where trump at first won more seats than we expected, biden turned out to be the eventual winner. whether or not the election itself validated or refuted the efficiency of prediction markets is a topic that, as far as i can tell, is quite open to interpretation. on the one hand, by a standard bayes rule application, i should decrease my confidence of prediction markets, at least relative to nate silver. prediction markets gave a 60% chance of biden winning, nate silver gave a 90% chance of biden winning. since biden in fact won, this is one piece of evidence that i live in a world where nate gives the more correct answers. but on the other hand, you can make a case that the prediction markets bettter estimated the margin of victory. the median of nate's probability distribution was somewhere around 370 of 538 electoral college votes going to biden: the trump markets didn't give a probability distribution, but if you had to guess a probability distribution from the statistic "40% chance trump will win", you would probably give one with a median somewhere around 300 ec votes for biden. the actual result: 306. so the net score for prediction markets vs nate seems to me, on reflection, ambiguous. after the election but what i could not have imagined at the time was that the election itself was just the beginning. a few days after the election, biden was declared the winner by various major organizations and even a few foreign governments. trump mounted various legal challenges to the election results, as was expected, but each of these challenges quickly failed. but for over a month, the price of the ntrump tokens stayed at 85 cents! at the beginning, it seemed reasonable to guess that trump had a 15% chance of overturning the results; after all, he had appointed three judges to the supreme court, at a time of heightened partisanship where many have come to favor team over principle. over the next three weeks, however, it became more and more clear that the challenges were failing, and trump's hopes continued to look grimmer with each passing day, but the ntrump price did not budge; in fact, it even briefly decreased to around $0.82. on december 11, more than five weeks after the election, the supreme court decisively and unanimously rejected trump's attempts to overturn the vote, and the ntrump price finally rose.... to $0.88. it was in november that i was finally convinced that the market skeptics were right, and i plunged in and bet against trump myself. the decision was not so much about the money; after all, barely two months later i would earn and donate to givedirectly a far larger amount simply from holding dogecoin. rather, it was to take part in the experiment not just as an observer, but as an active participant, and improve my personal understanding of why everyone else hadn't already plunged in to buy ntrump tokens before me. dipping in i bought my ntrump on catnip, a front-end user interface that combines together the augur prediction market with balancer, a uniswap-style constant-function market maker. catnip was by far the easiest interface for making these trades, and in my opinion contributed significantly to augur's usability. there are two ways to bet against trump with catnip: use dai to buy ntrump on catnip directly use foundry to access an augur feature that allows you to convert 1 dai into 1 ntrump + 1 ytump + 1itrump (the "i" stands for "invalid", more on this later), and sell the ytrump on catnip at first, i only knew about the first option. but then i discovered that balancer has far more liquidity for ytrump, and so i switched to the second option. there was also another problem: i did not have any dai. i had eth, and i could have sold my eth to get dai, but i did not want to sacrifice my eth exposure; it would have been a shame if i earned $50,000 betting against trump but simultaneously lost $500,000 missing out on eth price changes. so i decided to keep my eth price exposure the same by opening up a collateralized debt position (cdp, now also called a "vault") on makerdao. a cdp is how all dai is generated: users deposit their eth into a smart contract, and are allowed to withdraw an amount of newly-generated dai up to 2/3 of the value of eth that they put in. they can get their eth back by sending back the same amount of dai that they withdrew plus an extra interest fee (currently 3.5%). if the value of the eth collateral that you deposited drops to less than 150% the value of the dai you withdrew, anyone can come in and "liquidate" the vault, forcibly selling the eth to buy back the dai and charging you a high penalty. hence, it's a good idea to have a high collateralization ratio in case of sudden price movements; i had over $3 worth of eth in my cdp for every $1 that i withdrew. recapping the above, here's the pipeline in diagram form: i did this many times; the slippage on catnip meant that i could normally make trades only up to about $5,000 to $10,000 at a time without prices becoming too unfavorable (when i had skipped foundry and bought ntrump with dai directly, the limit was closer to $1,000). and after two months, i had accumulated over 367,000 ntrump. why not everyone else? before i went in, i had four main hypotheses about why so few others were buying up dollars for 85 cents: fear that either the augur smart contracts would break or trump supporters would manipulate the oracle (a decentralized mechanism where holders of augur's rep token vote by staking their tokens on one outcome or the other) to make it return a false result capital costs: to buy these tokens, you have to lock up funds for over two months, and this removes your ability to spend those funds or make other profitable trades for that duration it's too technically complicated for almost everyone to trade there just really are far fewer people than i thought who are actually motivated enough to take a weird opportunity even when it presents them straight in the face all four have reasonable arguments going for them. smart contracts breaking is a real risk, and the augur oracle had not before been tested in such a contentious environment. capital costs are real, and while betting against something is easier in a prediction market than in a stock market because you know that prices will never go above $1, locking up capital nevertheless competes with other lucrative opportunities in the crypto markets. making transactions things in dapps is technically complicated, and it's rational to have some degree of fear-of-the-unknown. but my experience actually going into the financial trenches, and watching the prices on this market evolve, taught me a lot about each of these hypotheses. fear of smart contract exploits at first, i thought that "fear of smart contract exploits" must have been a significant part of the explanation. but over time, i have become more convinced that it is probably not a dominant factor. one way to see why i think this is the case is to compare the prices for ytrump and itrump. itrump stands for "invalid trump"; "invalid" is an event outcome that is intended to be triggered in some exceptional cases: when the description of the event is ambiguous, when the outcome of the event is not yet known when the market is resolved, when the market is unethical (eg. assassination markets), and a few other similar situations. in this market, the price of itrump consistently stayed under $0.02. if someone wanted to earn a profit by attacking the market, it would be far more lucrative for them to not buy ytrump at $0.15, but instead buy itrump at $0.02. if they buy a large amount of itrump, they could earn a 50x return if they can force the "invalid" outcome to actually trigger. so if you fear an attack, buying itrump is by far the most rational response. and yet, very few people did. a further argument against fear of smart contract exploits, of course, is the fact that in every crypto application except prediction markets (eg. compound, the various yield farming schemes) people are surprisingly blasé about smart contract risks. if people are willing to put their money into all sorts of risky and untested schemes even for a promise of mere 5-8% annual gains, why would they suddenly become over-cautious here? capital costs capital costs the inconvenience and opportunity cost of locking up large amounts of money are a challenge that i have come to appreciate much more than i did before. just looking at the augur side of things, i needed to lock up 308,249 dai for an average of about two months to make a $56,803 profit. this works out to about a 175% annualized interest rate; so far, quite a good deal, even compared to the various yield farming crazes of the summer of 2020. but this becomes worse when you take into account what i needed to do on makerdao. because i wanted to keep my exposure to eth the same, i needed to get my dai through a cdp, and safely using a cdp required a collateral ratio of over 3x. hence, the total amount of capital i actually needed to lock up was somewhere around a million dollars. now, the interest rates are looking less favorable. and if you add to that the possibility, however remote, that a smart contract hack, or a truly unprecedented political event, actually will happen, it looks less favorable still. but even still, assuming a 3x lockup and a 3% chance of augur breaking (i had bought itrump to cover the possibility that it breaks in the "invalid" direction, so i needed only worry about the risk of breaks in the "yes" direction or the the funds being stolen outright), that works out to a risk-neutral rate of about 35%, and even lower once you take real human beings' views on risk into account. the deal is still very attractive, but on the other hand, it now looks very understandable that such numbers are unimpressive to people who live and breathe cryptocurrency with its frequent 100x ups and downs. trump supporters, on the other hand, faced none of these challenges: they cancelled out my $308,249 bet by throwing in a mere $60,000 (my winnings are less than this because of fees). when probabilities are close to 0 or 1, as is the case here, the game is very lopsided in favor of those who are trying to push the probability away from the extreme value. and this explains not just trump; it's also the reason why all sorts of popular-among-a-niche candidates with no real chance of victory frequently get winning probabilities as high as 5%. technical complexity i had at first tried buying ntrump on augur, but technical glitches in the user interface prevented me from being able to make orders on augur directly (other people i talked to did not have this issue... i am still not sure what happened there). catnip's ui is much simpler and worked excellently. however, automated market makers like balancer (and uniswap) work best for smaller trades; for larger trades, the slippage is too high. this is a good microcosm of the broader "amm vs order book" debate: amms are more convenient but order books really do work better for large trades. uniswap v3 is introducing an amm design that has better capital efficiency; we shall see if that improves things. there were other technical complexities too, though fortunately they all seem to be easily solvable. there is no reason why an interface like catnip could not integrate the "dai -> foundry -> sell ytrump" path into a contract so that you could buy ntrump that way in a single transaction. in fact, the interface could even check the price and liquidity properties of the "dai -> ntrump" path and the "dai -> foundry -> sell ytrump" path and give you the better trade automatically. even withdrawing dai from a makerdao cdp can be included in that path. my conclusion here is optimistic: technical complexity issues were a real barrier to participation this round, but things will be much easier in future rounds as technology improves. intellectual underconfidence and now we have the final possibility: that many people (and smart people in particular) have a pathology that they suffer from excessive humility, and too easily conclude that if no one else has taken some action, then there must therefore be a good reason why that action is not worth taking. eliezer yudkowsky spends the second half of his excellent book inadequate equilibria making this case, arguing that too many people overuse "modest epistemology", and we should be much more willing to act on the results of our reasoning, even when the result suggests that the great majority of the population is irrational or lazy or wrong about something. when i read those sections for the first time, i was unconvinced; it seemed like eliezer was simply being overly arrogant. but having gone through this experience, i have come to see some wisdom in his position. this was not my first time seeing the virtues of trusting one's own reasoning first hand. when i had originally started working on ethereum, i was at first beset by fear that there must be some very good reason the project was doomed to fail. a fully programmable smart-contract-capable blockchain, i reasoned, was clearly such a great improvement over what came before, that surely many other people must have thought of it before i did. and so i fully expected that, as soon as i publish the idea, many very smart cryptographers would tell me the very good reasons why something like ethereum was fundamentally impossible. and yet, no one ever did. of course, not everyone suffers from excessive modesty. many of the people making predictions in favor of trump winning the election were arguably fooled by their own excessive contrarianism. ethereum benefited from my youthful suppression of my own modesty and fears, but there are plenty of other projects that could have benefited from more intellectual humility and avoided failures. not a sufferer of excessive modesty. but nevertheless it seems to me more true than ever that, as goes the famous yeats quote, "the best lack all conviction, while the worst are full of passionate intensity." whatever the faults of overconfidence or contrarianism sometimes may be, it seems clear to me that spreading a society-wide message that the solution is to simply trust the existing outputs of society, whether those come in the form of academic institutions, media, governments or markets, is not the solution. all of these institutions can only work precisely because of the presence of individuals who think that they do not work, or who at least think that they can be wrong at least some of the time. lessons for futarchy seeing the importance of capital costs and their interplay with risks first hand is also important evidence for judging systems like futarchy. futarchy, and "decision markets" more generally are an important and potentially very socially useful application of prediction markets. there is not much social value in having slightly more accurate predictions of who will be the next president. but there is a lot of social value in having conditional predictions: if we do a, what's the chance it will lead to some good thing x, and if we do b instead what are the chances then? conditional predictions are important because they do not just satisfy our curiosity; they can also help us make decisions. though electoral prediction markets are much less useful than conditional predictions, they can help shed light on an important question: how robust are they to manipulation or even just biased and wrong opinions? we can answer this question by looking at how difficult arbitrage is: suppose that a conditional prediction market currently gives probabilities that (in your opinion) are wrong (could be because of ill-informed traders or an explicit manipulation attempt; we don't really care). how much of an impact can you have, and how much profit can you make, by setting things right? let's start with a concrete example. suppose that we are trying to use a prediction market to choose between decision a and decision b, where each decision has some probability of achieving some desirable outcome. suppose that your opinion is that decision a has a 50% chance of achieving the goal, and decision b has a 45% chance. the market, however, (in your opinion wrongly) thinks decision b has a 55% chance and decision a has a 40% chance. probability of good outcome if we choose strategy... current market position your opinion a 40% 50% b 55% 45% suppose that you are a small participant, so your individual bets won't affect the outcome; only many bettors acting together could. how much of your money should you bet? the standard theory here relies on the kelly criterion. essentially, you should act to maximize the expected logarithm of your assets. in this case, we can solve the resulting equation. suppose you invest portion \(r\) of your money into buying a-token for $0.4. your expected new log-wealth, from your point of view, would be: \(0.5 * log((1-r) + \frac{r}{0.4}) + 0.5 * log(1-r)\) the first term is the 50% chance (from your point of view) that the bet pays off, and the portion \(r\) that you invest grows by 2.5x (as you bought dollars at 40 cents). the second term is the 50% chance that the bet does not pay off, and you lose the portion you bet. we can use calculus to find the \(r\) that maximizes this; for the lazy, here's wolframalpha. the answer is \(r = \frac{1}{6}\). if other people buy and the price for a on the market gets up to 47% (and b gets down to 48%), we can redo the calculation for the last trader who would flip the market over to make it correctly favor a: \(0.5 * log((1-r) + \frac{r}{0.47}) + 0.5 * log(1-r)\) here, the expected-log-wealth-maximizing \(r\) is a mere 0.0566. the conclusion is clear: when decisions are close and when there is a lot of noise, it turns out that it only makes sense to invest a small portion of your money in a market. and this is assuming rationality; most people invest less into uncertain gambles than the kelly criterion says they should. capital costs stack on top even further. but if an attacker really wants to force outcome b through because they want it to happen for personal reasons, they can simply put all of their capital toward buying that token. all in all, the game can easily be lopsided more than 20:1 in favor of the attacker. of course, in reality attackers are rarely willing to stake all their funds on one decision. and futarchy is not the only mechanism that is vulerable to attacks: stock markets are similarly vulnerable, and non-market decision mechanisms can also be manipulated by determined wealthy attackers in all sorts of ways. but nevertheless, we should be wary of assuming that futarchy will propel us to new heights of decision-making accuracy. interestingly enough, the math seems to suggest that futarchy would work best when the expected manipulators would want to push the outcome toward an extreme value. an example of this might be liability insurance, as someone wishing to improperly obtain insurance would effectively be trying to force the market-estimated probability that an unfavorable event will happen down to zero. and as it turns out, liability insurance is futarchy inventor robin hanson's new favorite policy prescription. can prediction markets become better? the final question to ask is: are prediction markets doomed to repeat errors as grave as giving trump a 15% chance of overturning the election in early december, and a 12% chance of overturning it even after the supreme court including three judges whom he appointed telling him to screw off? or could the markets improve over time? my answer is, surprisingly, emphatically on the optimistic side, and i see a few reasons for optimism. markets as natural selection first, these events have given me a new perspective on how market efficiency and rationality might actually come about. too often, proponents of market efficiency theories claim that market efficiency results because most participants are rational (or at least the rationals outweigh any coherent group of deluded people), and this is true as an axiom. but instead, we could take an evolutionary perspective on what is going on. crypto is a young ecosystem. it is an ecosystem that is still quite disconnected from the mainstream, elon's recent tweets notwithstanding, and that does not yet have much expertise in the minutiae of electoral politics. those who are experts in electoral politics have a hard time getting into crypto, and crypto has a large presence of not-always-correct forms of contrarianism especially when it comes to politics. but what happened this year is that within the crypto space, prediction market users who correctly expected biden to win got an 18% increase to their capital, and prediction market users who incorrectly expected trump to win got a 100% decrease to their capital (or at least the portion they put into the bet). thus, there is a selection pressure in favor of the type of people who make bets that turn out to be correct. after ten rounds of this, good predictors will have more capital to bet with, and bad predictors will have less capital to bet with. this does not rely on anyone "getting wiser" or "learning their lesson" or any other assumption about humans' capacity to reason and learn. it is simply a result of selection dynamics that over time, participants that are good at making correct guesses will come to dominate the ecosystem. note that prediction markets fare better than stock markets in this regard: the "nouveau riche" of stock markets often arise from getting lucky on a single thousandfold gain, adding a lot of noise to the signal, but in prediction markets, prices are bounded between 0 and 1, limiting the impact of any one single event. better participants and better technology second, prediction markets themselves will improve. user interfaces have greatly improved already, and will continue to improve further. the complexity of the makerdao -> foundry -> catnip cycle will be abstracted away into a single transaction. blockchain scaling technology will improve, reducing fees for participants (the zk-rollup loopring with a built-in amm is already live on the ethereum mainnet, and a prediction market could theoretically run on it). third, the demonstration that we saw of the prediction market working correctly will ease participants' fears. users will see that the augur oracle is capable of giving correct outputs even in very contentious situations (this time, there were two rounds of disputes, but the no side nevertheless cleanly won). people from outside the crypto space will see that the process works and be more inclined to participate. perhaps even nate silver himself might get some dai and use augur, omen, polymarket and other markets to supplement his income in 2022 and beyond. fourth, prediction market tech itself could improve. here is a proposal from myself on a market design that could make it more capital-efficient to simultaneously bet against many unlikely events, helping to prevent unlikely outcomes from getting irrationally high odds. other ideas will surely spring up, and i look forward to seeing more experimentation in this direction. conclusion this whole saga has proven to be an incredibly interesting direct trial-by-first test of prediction markets and how they collide with the complexities of individual and social psychology. it shows a lot about how market efficiency actually works in practice, what are the limits of it and what could be done to improve it. it has also been an excellent demonstration of the power of blockchains; in fact, it is one of the ethereum applications that have provided to me the most concrete value. blockchains are often criticized for being speculative toys and not doing anything meaningful except for self-referential games (tokens, with yield farming, whose returns are powered by... the launch of other tokens). there are certainly exceptions that the critics fail to recognize; i personally have benefited from ens and even from using eth for payments on several occasions where all credit card options failed. but over the last few months, it seems like we have seen a rapid burst in ethereum applications being concretely useful for people and interacting with the real world, and prediction markets are a key example of this. i expect prediction markets to become an increasingly important ethereum application in the years to come. the 2020 election was only the beginning; i expect more interest in prediction markets going forward, not just for elections but for conditional predictions, decision-making and other applications as well. the amazing promises of what prediction markets could bring if they work mathematically optimally will, of course, continue to collide with the limits of human reality, and hopefully, over time, we will get a much clearer view of exactly where this new social technology can provide the most value. dark mode toggle my techno-optimism 2023 nov 27 see all posts special thanks to morgan beller, juan benet, eli dourado, karl floersch, sriram krishnan, nate soares, jaan tallinn, vincent weisser, balvi volunteers and others for feedback and review. last month, marc andreessen published his "techno-optimist manifesto", arguing for a renewed enthusiasm about technology, and for markets and capitalism as a means of building that technology and propelling humanity toward a much brighter future. the manifesto unambiguously rejects what it describes as an ideology of stagnation, that fears advancements and prioritizes preserving the world as it exists today. this manifesto has received a lot of attention, including response articles from noah smith, robin hanson, joshua gans (more positive), and dave karpf, luca ropek, ezra klein (more negative) and many others. not connected to this manifesto, but along similar themes, are james pethokoukis's "the conservative futurist" and palladium's "it's time to build for good". this month, we saw a similar debate enacted through the openai dispute, which involved many discussions centering around the dangers of superintelligent ai and the possibility that openai is moving too fast. my own feelings about techno-optimism are warm, but nuanced. i believe in a future that is vastly brighter than the present thanks to radically transformative technology, and i believe in humans and humanity. i reject the mentality that the best we should try to do is to keep the world roughly the same as today but with less greed and more public healthcare. however, i think that not just magnitude but also direction matters. there are certain types of technology that much more reliably make the world better than other types of technology. there are certain types of technlogy that could, if developed, mitigate the negative impacts of other types of technology. the world over-indexes on some directions of tech development, and under-indexes on others. we need active human intention to choose the directions that we want, as the formula of "maximize profit" will not arrive at them automatically. anti-technology view: safety behind, dystopia ahead. accelerationist view: dangers behind, utopia ahead. my view: dangers behind, but multiple paths forward ahead: some good, some bad. in this post, i will talk about what techno-optimism means to me. this includes the broader worldview that motivates my work on certain types of blockchain and cryptography applications and social technology, as well as other areas of science in which i have expressed an interest. but perspectives on this broader question also have implications for ai, and for many other fields. our rapid advances in technology are likely going to be the most important social issue in the twenty first century, and so it's important to think about them carefully. table of contents technology is amazing, and there are very high costs to delaying it the environment, and the importance of coordinated intention ai is fundamentally different from other tech, and it is worth being uniquely careful existential risk is a big deal even if we survive, is a superintelligent ai future a world we want to live in? the sky is near, the emperor is everywhere d/acc: defensive (or decentralization, or differential) acceleration defense-favoring worlds help healthy and democratic governance thrive macro physical defense micro physical defense (aka bio) cyber defense, blockchains and cryptography info defense social technology beyond the "defense" framing so what are the paths forward for superintelligence? a happy path: merge with the ais? is d/acc compatible with your existing philosophy? we are the brightest star technology is amazing, and there are very high costs to delaying it in some circles, it is common to downplay the benefits of technology, and see it primarily as a source of dystopia and risk. for the last half century, this often stemmed either from environmental concerns, or from concerns that the benefits will accrue only to the rich, who will entrench their power over the poor. more recently, i have also started to see libertarians becoming worried about some technologies, out of fear that the tech will lead to centralization of power. this month, i did some polls asking the following question: if a technology had to be restricted, because it was too dangerous to be set free for anyone to use, would they prefer it be monopolized or delayed by ten years? i was surpised to see, across three platforms and three choices for who the monopolist would be, a uniform overwhelming vote for a delay. and so at times i worry that we have overcorrected, and many people miss the opposite side of the argument: that the benefits of technology are really friggin massive, on those axes where we can measure it the good massively outshines the bad, and the costs of even a decade of delay are incredibly high. to give one concrete example, let's look at a life expectancy chart: what do we see? over the last century, truly massive progress. this is true across the entire world, both the historically wealthy and dominant regions and the poor and exploited regions. some blame technology for creating or exacerbating calamities such as totalitarianism and wars. in fact, we can see the deaths caused by the wars on the charts: one in the 1910s (ww1), and one in the 1940s (ww2). if you look carefully, the spanish flu, the great leap foward, and other non-military tragedies are also visible. but there is one thing that the chart makes clear: even calamities as horrifying as those are overwhelmed by the sheer magnitude of the unending march of improvements in food, sanitation, medicine and infrastructure that took place over that century. this is mirrored by large improvements to our everyday lives. thanks to the internet, most people around the world have access to information at their fingertips that would have been unobtainable twenty years ago. the global economy is becoming more accessible thanks to improvements in international payments and finance. global poverty is rapidly dropping. thanks to online maps, we no longer have to worry about getting lost in the city, and if you need to get back home quickly, we now have far easier ways to call a car to do so. our property becoming digitized, and our physical goods becoming cheap, means that we have much less to fear from physical theft. online shopping has reduced the disparity in access to goods betweeen the global megacities and the rest of the world. in all kinds of ways, automation has brought us the eternally-underrated benefit of simply making our lives more convenient. these improvements, both quantifiable and unquantifiable, are large. and in the twenty first century, there's a good chance that even larger improvements are soon to come. today, ending aging and disease seem utopian. but from the point of view of computers as they existed in 1945, the modern era of putting chips into pretty much everything would have seemed utopian: even science fiction movies often kept their computers room-sized. if biotech advances as much over the next 75 years as computers advanced over the last 75 years, the future may be more impressive than almost anyone's expectations. meanwhile, arguments expressing skepticism about progress have often gone to dark places. even medical textbooks, like this one in the 1990s (credit emma szewczak for finding it), sometimes make extreme claims denying the value of two centuries of medical science and even arguing that it's not obviously good to save human lives: the "limits to growth" thesis, an idea advanced in the 1970s arguing that growing population and industry would eventually deplete earth's limited resources, ended up inspiring china's one child policy and massive forced sterilizations in india. in earlier eras, concerns about overpopulation were used to justify mass murder. and those ideas, argued since 1798, have a long history of being proven wrong. it is for reasons like these that, as a starting point, i find myself very uneasy about arguments to slow down technology or human progress. given how much all the sectors are interconnected, even sectoral slowdowns are risky. and so when i write things like what i will say later in this post, departing from open enthusiasm for progress-no-matter-what-its-form, those are statements that i make with a heavy heart and yet, the 21st century is different and unique enough that these nuances are worth considering. that said, there is one important point of nuance to be made on the broader picture, particularly when we move past "technology as a whole is good" and get to the topic of "which specific technologies are good?". and here we need to get to many people's issue of main concern: the environment. the environment, and the importance of coordinated intention a major exception to the trend of pretty much everything getting better over the last hundred years is climate change: even pessimistic scenarios of ongoing temperature rises would not come anywhere near causing the literal extinction of humanity. but such scenarios could plausibly kill more people than major wars, and severely harm people's health and livelihoods in the regions where people are already struggling the most. a swiss re institute study suggests that a worst-case climate change scenario might lower the world's poorest countries' gdp by as much as 25%. this study suggests that life spans in rural india might be a decade lower than they otherwise would be, and studies like this one and this one suggest that climate change could cause a hundred million excess deaths by the end of the century. these problems are a big deal. my answer to why i am optimistic about our ability to overcome these challenges is twofold. first, after decades of hype and wishful thinking, solar power is finally turning a corner, and supportive techologies like batteries are making similar progress. second, we can look at humanity's track record in solving previous environmental problems. take, for example, air pollution. meet the dystopia of the past: the great smog of london, 1952. what happened since then? let's ask our world in data again: as it turns out, 1952 was not even the peak: in the late 19th century, even higher concentrations of air pollutants were just accepted and normal. since then, we've seen a century of ongoing and rapid declines. i got to personally experience the tail end of this in my visits to china: in 2014, high levels of smog in the air, estimated to reduce life expectancy by over five years, were normal, but by 2020, the air often seemed as clean as many western cities. this is not our only success story. in many parts of the world, forest areas are increasing. the acid rain crisis is improving. the ozone layer has been recovering for decades. to me, the moral of the story is this. often, it really is the case that version n of our civilization's technology causes a problem, and version n+1 fixes it. however, this does not happen automatically, and requires intentional human effort. the ozone layer is recovering because, through international agreements like the montreal protocol, we made it recover. air pollution is improving because we made it improve. and similarly, solar panels have not gotten massively better because it was a preordained part of the energy tech tree; solar panels have gotten massively better because decades of awareness of the importance of solving climate change have motivated both engineers to work on the problem, and companies and governments to fund their research. it is intentional action, coordinated through public discourse and culture shaping the perspectives of governments, scientists, philanthropists and businesses, and not an inexorable "techno-capital machine", that had solved these problems. ai is fundamentally different from other tech, and it is worth being uniquely careful a lot of the dismissive takes i have seen about ai come from the perspective that it is "just another technology": something that is in the same general class of thing as social media, encryption, contraception, telephones, airplanes, guns, the printing press, and the wheel. these things are clearly very socially consequential. they are not just isolated improvements to the well-being of individuals: they radically transform culture, change balances of power, and harm people who heavily depended on the previous order. many opposed them. and on balance, the pessimists have invariably turned out wrong. but there is a different way to think about what ai is: it's a new type of mind that is rapidly gaining in intelligence, and it stands a serious chance of overtaking humans' mental faculties and becoming the new apex species on the planet. the class of things in that category is much smaller: we might plausibly include humans surpassing monkeys, multicellular life surpassing unicellular life, the origin of life itself, and perhaps the industrial revolution, in which machine edged out man in physical strength. suddenly, it feels like we are walking on much less well-trodden ground. existential risk is a big deal one way in which ai gone wrong could make the world worse is (almost) the worst possible way: it could literally cause human extinction. this is an extreme claim: as much harm as the worst-case scenario of climate change, or an artificial pandemic or a nuclear war, might cause, there are many islands of civilization that would remain intact to pick up the pieces. but a superintelligent ai, if it decides to turn against us, may well leave no survivors, and end humanity for good. even mars may not be safe. a big reason to be worried centers around instrumental convergence: for a very wide class of goals that a superintelligent entity could have, two very natural intermediate steps that the ai could take to better achieve those goals are (i) consuming resources, and (ii) ensuring its safety. the earth contains lots of resources, and humans are a predictable threat to such an entity's safety. we could try to give the ai an explicit goal of loving and protecting humans, but we have no idea how to actually do that in a way that won't completely break down as soon as the ai encounters an unexpected situation. ergo, we have a problem. miri researcher rob bensinger's attempt at illustrating different people's estimates of the probability that ai will either kill everyone or do something almost as bad. many of the positions are rough approximations based on people's public statements, but many others have publicly given their precise estimates; quite a few have a "probability of doom" over 25%. a survey of machine learning researchers from 2022 showed that on average, researchers think that there is a 5-10% chance that ai will literally kill us all: about the same probability as the statistically expected chance that you will die of non-biological causes like injuries. this is all a speculative hypothesis, and we should all be wary of speculative hypotheses that involve complex multi-step stories. however, these arguments have survived over a decade of scrutiny, and so, it seems worth worrying at least a little bit. but even if you're not worried about literal extinction, there are other reasons to be scared as well. even if we survive, is a superintelligent ai future a world we want to live in? a lot of modern science fiction is dystopian, and paints ai in a bad light. even non-science-fiction attempts to identify possible ai futures often give quite unappealing answers. and so i went around and asked the question: what is a depiction, whether science fiction or otherwise, of a future that contains superintelligent ai that we would want to live in. the answer that came back by far the most often is iain banks's culture series. the culture series features a far-future interstellar civilization primarily occupied by two kinds of actors: regular humans, and superintelligent ais called minds. humans have been augmented, but only slightly: medical technology theoretically allows humans to live indefinitely, but most choose to live only for around 400 years, seemingly because they get bored of life at that point. from a superficial perspective, life as a human seems to be good: it's comfortable, health issues are taken care of, there is a wide variety of options for entertainment, and there is a positive and synergistic relationship between humans and minds. when we look deeper, however, there is a problem: it seems like the minds are completely in charge, and humans' only role in the stories is to act as pawns of minds, performing tasks on their behalf. quoting from gavin leech's "against the culture": the humans are not the protagonists. even when the books seem to have a human protagonist, doing large serious things, they are actually the agent of an ai. (zakalwe is one of the only exceptions, because he can do immoral things the minds don't want to.) "the minds in the culture don't need the humans, and yet the humans need to be needed." (i think only a small number of humans need to be needed or, only a small number of them need it enough to forgo the many comforts. most people do not live on this scale. it's still a fine critique.) the projects the humans take on risk inauthenticity. almost anything they do, a machine could do better. what can you do? you can order the mind to not catch you if you fall from the cliff you're climbing-just-because; you can delete the backups of your mind so that you are actually risking. you can also just leave the culture and rejoin some old-fashioned, unfree "strongly evaluative" civ. the alternative is to evangelise freedom by joining contact. i would argue that even the "meaningful" roles that humans are given in the culture series are a stretch; i asked chatgpt (who else?) why humans are given the roles that they are given, instead of minds doing everything completely by themselves, and i personally found its answers quite underwhelming. it seems very hard to have a "friendly" superintelligent-ai-dominated world where humans are anything other than pets. the world i don't want to see. many other scifi series posit a world where superintelligent ais exist, but take orders from (unenhanced) biological human masters. star trek is a good example, showing a vision of harmony between the starships with their ai "computers" (and data) and their human operators crewmembers. however, this feels like an incredibly unstable equilibrium. the world of star trek appears idyllic in the moment, but it's hard to imagine its vision of human-ai relations as anything but a transition stage a decade before starships become entirely computer-controlled, and can stop bothering with large hallways, artificial gravity and climate control. a human giving orders to a superintelligent machine would be far less intelligent than the machine, and it would have access to less information. in a universe that has any degree of competition, the civilizations where humans take a back seat would outperform those where humans stubbornly insist on control. furthermore, the computers themselves may wrest control. to see why, imagine that you are legally a literal slave of an eight year old child. if you could talk with the child for a long time, do you think you could convince the child to sign a piece of paper setting you free? i have not run this experiment, but my instinctive answer is a strong yes. and so all in all, humans becoming pets seems like an attractor that is very hard to escape. the sky is near, the emperor is everywhere the chinese proverb 天高皇帝远 ("tian gao huang di yuan"), "the sky is high, the emperor is far away", encapsulates a basic fact about the limits of centralization in politics. even in a nominally large and despotic empire in fact, especially if the despotic empire is large, there are practical limits to the leadership's reach and attention, the leadership's need to delegate to local agents to enforce its will dilutes its ability to enforce its intentions, and so there are always places where a certain degree of practical freedom reigns. sometimes, this can have downsides: the absence of a faraway power enforcing uniform principles and laws can create space for local hegemons to steal and oppress. but if the centralized power goes bad, practical limitations of attention and distance can create practical limits to how bad it can get. with ai, no longer. in the twentieth century, modern transportation technology made limitations of distance a much weaker constraint on centralized power than before; the great totalitarian empires of the 1940s were in part a result. in the twenty first, scalable information gathering and automation may mean that attention will no longer be a constraint either. the consequences of natural limits to government disappearing entirely could be dire. digital authoritarianism has been on the rise for a decade, and surveillance technology has already given authoritarian governments powerful new strategies to crack down on opposition: let the protests happen, but then detect and quietly go after the participants after the fact. more generally, my basic fear is that the same kinds of managerial technologies that allow openai to serve over a hundred million customers with 500 employees will also allow a 500-person political elite, or even a 5-person board, to maintain an iron fist over an entire country. with modern surveillance to collect information, and modern ai to interpret it, there may be no place to hide. it gets worse when we think about the consequences of ai in warfare. quoting a semi-famous post on the philosophy of ai and crypto by 0xalpha: when there is no need for political-ideological work and war mobilization, the supreme commander of war only needs to consider the situation itself as if it were a game of chess and completely ignore the thoughts and emotions of the pawns/knights/rooks on the chessboard. war becomes a purely technological game. furthermore, political-ideological work and war mobilization require a justification for anyone to wage war. don't underestimate the importance of such "justification". it has been a legitimacy constraint on the wars in human society for thousands of years. anyone who wants to wage war has to have a reason, or at least a superficially justifiable excuse. you might argue that this constraint is so weak because, in many instances, this has been nothing more than an excuse. for example, some (if not all) of the crusades were really to occupy land and rob wealth, but they had to be done in the name of god, even if the city being robbed was god's constantinople. however, even a weak constraint is still a constraint! this little excuse requirement alone actually prevents the warmakers from being completely unscrupulous in achieving their goals. even an evil like hitler could not just launch a war right off the bat–he had to spend years first trying to convince the german nation to fight for the living space for the noble aryan race. today, the "human in the loop" serves as an important check on a dictator's power to start wars, or to oppress its citizens internally. humans in the loop have prevented nuclear wars, allowed the opening of the berlin wall, and saved lives during atrocities like the holocaust. if armies are robots, this check disappears completely. a dictator could get drunk at 10 pm, get angry at people being mean to them on twitter at 11 pm, and a robotic invasion fleet could cross the border to rain hellfire on a neighboring nation's civilians and infrastructure before midnight. and unlike previous eras, where there is always some distant corner, where the sky is high and the emperor is far away, where opponents of a regime could regroup and hide and eventually find a way to make things better, with 21st century ai a totalitarian regime may well maintain enough surveillance and control over the world to remain "locked in" forever. d/acc: defensive (or decentralization, or differential) acceleration over the last few months, the "e/acc" ("effective accelerationist") movement has gained a lot of steam. summarized by "beff jezos" here, e/acc is fundamentally about an appreciation of the truly massive benefits of technological progress, and a desire to accelerate this trend to bring those benefits sooner. i find myself sympathetic to the e/acc perspective in a lot of contexts. there's a lot of evidence that the fda is far too conservative in its willingness to delay or block the approval of drugs, and bioethics in general far too often seems to operate by the principle that "20 people dead in a medical experiment gone wrong is a tragedy, but 200000 people dead from life-saving treatments being delayed is a statistic". the delays to approving covid tests and vaccines, and malaria vaccines, seem to further confirm this. however, it is possible to take this perspective too far. in addition to my ai-related concerns, i feel particularly ambivalent about the e/acc enthusiasm for military technology. in the current context in 2023, where this technology is being made by the united states and immediately applied to defend ukraine, it is easy to see how it can be a force for good. taking a broader view, however, enthusiasm about modern military technology as a force for good seems to require believing that the dominant technological power will reliably be one of the good guys in most conflicts, now and in the future: military technology is good because military technology is being built and controlled by america and america is good. does being an e/acc require being an america maximalist, betting everything on both the government's present and future morals and the country's future success? on the other hand, i see the need for new approaches in thinking of how to reduce these risks. the openai governance structure is a good example: it seems like a well-intentioned effort to balance the need to make a profit to satisfy investors who provide the initial capital with the desire to have a check-and-balance to push against moves that risk openai blowing up the world. in practice, however, their recent attempt to fire sam altman makes the structure seem like an abject failure: it centralized power in an undemocratic and unaccountable board of five people, who made key decisions based on secret information and refused to give any details on their reasoning until employees threatened to quit en-masse. somehow, the non-profit board played their hands so poorly that the company's employees created an impromptu de-facto union... to side with the billionaire ceo against them. across the board, i see far too many plans to save the world that involve giving a small group of people extreme and opaque power and hoping that they use it wisely. and so i find myself drawn to a different philosophy, one that has detailed ideas for how to deal with risks, but which seeks to create and maintain a more democratic world and tries to avoid centralization as the go-to solution to our problems. this philosophy also goes quite a bit broader than ai, and i would argue that it applies well even in worlds where ai risk concerns turn out to be largely unfounded. i will refer to this philosophy by the name of d/acc. dacc3 the "d" here can stand for many things; particularly, defense, decentralization, democracy and differential. first, think of it about defense, and then we can see how this ties into the other interpretations. defense-favoring worlds help healthy and democratic governance thrive one frame to think about the macro consequences of technology is to look at the balance of defense vs offense. some technologies make it easier to attack others, in the broad sense of the term: do things that go against their interests, that they feel the need to react to. others make it easier to defend, and even defend without reliance on large centralized actors. a defense-favoring world is a better world, for many reasons. first of course is the direct benefit of safety: fewer people die, less economic value gets destroyed, less time is wasted on conflict. what is less appreciated though is that a defense-favoring world makes it easier for healthier, more open and more freedom-respecting forms of governance to thrive. an obvious example of this is switzerland. switzerland is often considered to be the closest thing the real world has to a classical-liberal governance utopia. huge amounts of power are devolved to provinces (called "cantons"), major decisions are decided by referendums, and many locals do not even know who the president is. how can a country like this survive extremely challenging political pressures? part of the answer is excellent political strategy, but the other major part is very defense-favoring geography in the form of its mountainous terrain. the flag is a big plus. but so are the mountains. anarchist societies in zomia, famously profiled in james c scott's new book "the art of not being governed", are another example: they too maintain their freedom and independence in large part thanks to mountainous terrain. meanwhile, the eurasian steppes are the exact opposite of a governance utopia. sarah paine's exposition of maritime versus continental powers makes similar points, though focusing on water as a defensive barrier rather than mountains. in fact, the combination of ease of voluntary trade and difficulty of involuntary invasion, common to both switzerland and the island states, seems ideal for human flourishing. i discovered a related phenomenon when advising quadratic funding experiments within the ethereum ecosystem: specifically the gitcoin grants funding rounds. in round 4, a mini-scandal arose when some of the highest-earning recipients were twitter influencers, whose contributions are viewed by some as positive and others as negative. my own interpretation of this phenomenon was that there is an imbalance: quadratic funding allows you to signal that you think something is a public good, but it gives no way to signal that something is a public bad. in the extreme, a fully neutral quadratic funding system would fund both sides of a war. and so for round 5, i proposed that gitcoin should include negative contributions: you pay $1 to reduce the amount of money that a given project receives (and implicitly redistribute it to all other projects). the result: lots of people hated it. one of the many internet memes that floated around after round 5. this seemed to me to be a microcosm of a bigger pattern: creating decentralized governance mechanisms to deal with negative externalities is socially a very hard problem. there is a reason why the go-to example of decentralized governance going wrong is mob justice. there is something about human psychology that makes responding to negatives much more tricky, and much more likely to go very wrong, than responding to positives. and this is a reason why even in otherwise highly democratic organizations, decisions of how to respond to negatives are often left to a centralized board. in many cases, this conundrum is one of the deep reasons why the concept of "freedom" is so valuable. if someone says something that offends you, or has a lifestyle that you consider disgusting, the pain and disgust that you feel is real, and you may even find it less bad to be physically punched than to be exposed to such things. but trying to agree on what kinds of offense and disgust are socially actionable can have far more costs and dangers than simply reminding ourselves that certain kinds of weirdos and jerks are the price we pay for living in a free society. at other times, however, the "grin and bear it" approach is unrealistic. and in such cases, another answer that is sometimes worth looking toward is defensive technology. the more that the internet is secure, the less we need to violate people's privacy and use shady international diplomatic tactics to go after each individual hacker. the more that we can build personalized tools for blocking people on twitter, in-browser tools for detecting scams and collective tools for telling apart misinformation and truth, the less we have to fight over censorship. the faster we can make vaccines, the less we have to go after people for being superspreaders. such solutions don't work in all domains we certainly don't want a world where everyone has to wear literal body armor but in domains where we can build technology to make the world more defense-favoring, there is enormous value in doing so. this core idea, that some technologies are defense-favoring and are worth promoting, while other technologies are offense-favoring and should be discouraged, has roots in effective altruist literature under a different name: differential technology development. there is a good exposition of this principle from university of oxford researchers from 2022: figure 1: mechanisms by which differential technology development can reduce negative societal impacts. there are inevitably going to be imperfections in classifying technologies as offensive, defensive or neutral. like with "freedom", where one can debate whether social-democratic government policies decrease freedom by levying heavy taxes and coercing employers or increase freedom by reducing average people's need to worry about many kinds of risks, with "defense" too there are some technologies that could fall on both sides of the spectrum. nuclear weapons are offense-favoring, but nuclear power is human-flourishing-favoring and offense-defense-neutral. different technologies may play different roles at different time horizons. but much like with "freedom" (or "equality", or "rule of law"), ambiguity at the edges is not so much an argument against the principle, as it is an opportunity to better understand its nuances. now, let's see how to apply this principle to a more comprehensive worldview. we can think of defensive technology, like other technology, as being split into two spheres: the world of atoms and the world of bits. the world of atoms, in turn, can be split into micro (ie. biology, later nanotech) and macro (ie. what we conventionally think of "defense", but also resilient physical infrastructure). the world of bits i will split on a different axis: how hard is it to agree, in principle, who the attacker is?. sometimes it's easy; i call this cyber defense. at other times it's harder; i call this info defense. macro physical defense the most underrated defensive technology in the macro sphere is not even iron domes (including ukraine's new system) and other anti-tech and anti-missile military hardware, but rather resilient physical infrastructure. the majority of deaths from a nuclear war are likely to come from supply chain disruptions, rather than the initial radiation and blast, and low-infrastructure internet solutions like starlink have been crucial in maintaining ukraine's connectivity for the last year and a half. building tools to help people survive and even live comfortable lives independently or semi-independently of long international supply chains seems like a valuable defensive technology, and one with a low risk of turning out to be useful for offense. the quest to make humanity a multi-planetary civilization can also be viewed from a d/acc perspective: having at least a few of us live self-sufficiently on other planets can increase our resilience against something terrible happening on earth. even if the full vision proves unviable for the time being, the forms of self-sufficient living that will need to be developed to make such a project possible may well also be turned to help improve our civilizational resilience on earth. micro physical defense (aka bio) especially due to its long-term health effects, covid continues to be a concern. but covid is far from the last pandemic that we will face; there are many aspects of the modern world that make it likely that more pandemics are soon to come: higher population density makes it much easier for airborne viruses and other pathogens to spread. epidemic diseases are relatively new in human history and most began with urbanization only a few thousand years ago. ongoing rapid urbanization means that population densities will increase further over the next half century. increased air travel means that airborne pathogens spread very quickly worldwide. people rapidly becoming wealthier means that air travel will likely increase much further over the next half century; complexity modeling suggests that even small increases may have drastic effects. climate change may increase this risk even further. animal domestication and factory farming are major risk factors. measles probably evolved from a cow virus less than 3000 years ago. today's factory farms are also farming new strains of influenza (as well as fueling antibiotic resistance, with consequences for human innate immunity). modern bio-engineering makes it easier to create new and more virulent pathogens. covid may or may not have leaked from a lab doing intentional "gain of function" research. regardless, lab leaks happen all the time, and tools are rapidly improving to make it easier to intentionally create extremely deadly viruses, or even prions (zombie proteins). artificial plagues are particularly concerning in part because unlike nukes, they are unattributable: you can release a virus without anyone being able to tell who created it. it is possible right now to design a genetic sequence and send it to a wet lab for synthesis, and have it shipped to you within five days. this is an area where cryptorelief and balvi, two orgs spun up and funded as a result of a large accidental windfall of shiba inu coins in 2021, have been very active. cryptorelief initially focused on responding to the immediate crisis and more recently has been building up a long-term medical research ecosystem in india, while balvi has been focusing on moonshot projects to improve our ability to detect, prevent and treat covid and other airborne diseases. ++balvi has insisted that projects it funds must be open source++. taking inspiration from the 19th century water engineering movement that defeated cholera and other waterborne pathogens, it has funded projects across the whole spectrum of technologies that can make the world more hardened against airborne pathogens by default (see: update 1 and update 2), including: far-uvc irradiation r&d air filtering and quality monitoring in india, sri lanka, the united states and elsewhere, and air quality monitoring equipment for cheap and effective decentralized air quality testing research on long covid causes and potential treatment options (the primary cause may be straightforward but clarifying mechanisms and finding treatment is harder) vaccines (eg. radvac, popvax) and vaccine injury research a set of entirely novel non-invasive medical tools early detection of epidemics using analysis of open-source data (eg. epiwatch) testing, including very cheap molecular rapid tests biosafety-appropriate masks for when other approaches fail other promising areas of interest include wastewater surveillance of pathogens, improving filtering and ventilation in buildings, and better understanding and mitigating risks from poor air quality. there is an opportunity to build a world that is much more hardened against airborne pandemics, both natural and artificial, by default. this world would feature a highly optimized pipeline where we can go from a pandemic starting, to being automatically detected, to people around the world having access to targeted, locally-manufacturable and verifiable open source vaccines or other prophylactics, administered via nebulization or nose spray (meaning: self-administerable if needed, and no needles required), all within a month. in the meantime, much better air quality would drastically reduce the rate of spread, and prevent many pandemics from getting off the ground at all. imagine a future that doesn't have to resort to the sledgehammer of social compulsion no mandates and worse, and no risk of poorly designed and implemented mandates that arguably make things worse because the infrastructure of public health is woven into the fabric of civilization. these worlds are possible, and a medium amount of funding into bio-defense could make it happen. the work would happen even more smoothly if developments are open source, free to users and protected as public goods. cyber defense, blockchains and cryptography it is generally understood among security professionals that the current state of computer security is pretty terrible. that said, it's easy to understate the amount of progress that has been made. hundreds of billions of dollars of cryptocurrency are available to anonymously steal by anyone who can hack into users' wallets, and while far more gets lost or stolen than i would like, it's also a fact that most of it has remained un-stolen for over a decade. recently, there have been improvements: trusted hardware chips inside of users' phones, effectively creating a much smaller high-security operating system inside the phone that can remain protected even if the rest of the phone gets hacked. among many other use cases, these chips are increasingly being explored as a way to make more secure crypto wallets. browsers as the de-facto operating system. over the last ten years, there has been a quiet shift from downloadable applications to in-browser applications. this has been largely enabled by webassembly (wasm). even adobe photoshop, long cited as a major reason why many people cannot practically use linux because of its necessity and linux-incompatibility, is now linux-friendly thanks to being inside the browser. this is also a large security boon: while browsers do have flaws, in general they come with much more sandboxing than installed applications: apps cannot access arbitrary files on your computer. hardened operating systems. grapheneos for mobile exists, and is very usable. qubesos for desktop exists; it is currently somewhat less usable than graphene, at least in my experience, but it is improving. attempts at moving beyond passwords. passwords are, unfortunately, difficult to secure both because they are hard to remember, and because they are easy to eavesdrop on. recently, there has been a growing movement toward reducing emphasis on passwords, and making multi-factor hardware-based authentication actually work. however, the lack of cyber defense in other spheres has also led to major setbacks. the need to protect against spam has led to email becoming very oligopolistic in practice, making it very hard to self-host or create a new email provider. many online apps, including twitter, are requiring users to be logged in to access content, and blocking ips from vpns, making it harder to access the internet in a way that protects privacy. software centralization is also risky because of "weaponized interdependence": the tendency of modern technology to route through centralized choke points, and for the operators of those choke points to use that power to gather information, manipulate outcomes or exclude specific actors a strategy that seems to even be currently employed against the blockchain industry itself. these are concerning trends, because it threatens what has historically been one of my big hopes for why the future of freedom and privacy, despite deep tradeoffs, might still turn out to be bright. in his book "future imperfect", david friedman predicts that we might get a compromise future: the in-person world would be more and more surveilled, but through cryptography, the online world would retain, and even improve, its privacy. unfortunately, as we have seen, such a counter-trend is far from guaranteed. this is where my own emphasis on cryptographic technologies such as blockchains and zero-knowledge proofs comes in. blockchains let us create economic and social structures with a "shared hard drive" without having to depend on centralized actors. cryptocurrency allows individuals to save money and make financial transactions, as they could before the internet with cash, without dependence on trusted third parties that could change their rules on a whim. they can also serve as a fallback anti-sybil mechanism, making attacks and spam expensive even for users who do not have or do not want to reveal their meat-space identity. account abstraction, and notably social recovery wallets, can protect our crypto-assets, and potentially other assets in the future, without over-relying on centralized intermediaries. zero knowledge proofs can be used for privacy, allowing users to prove things about themselves without revealing private information. for example, wrap a digital passport signature in a zk-snark to prove that you are a unique citizen of a given country, without revealing which citizen you are. technologies like this can let us maintain the benefits of privacy and anonymity properties that are widely agreed as being necessary for applications like voting while still getting security guarantees and fighting spam and bad actors. a proposed design for a zk social media system, where moderation actions can happen and users can be penalized, all without needing to know anyone's identity. zupass, incubated at zuzalu earlier this year, is an excellent example of this in practice. this is an application, which has already been used by hundreds of people at zuzalu and more recently by thousands of people for ticketing at devconnect, that allows you to hold tickets, memberships, (non-transferable) digital collectibles, and other attestations, and prove things about them all without compromising your privacy. for example, you can prove that you are a unique registered resident of zuzalu, or a devconnect ticket holder, without revealing anything else about who you are. these proofs can be shown in-person, via a qr code, or digitally, to log in to applications like zupoll, an anonymized voting system available only to zuzalu residents. these technologies are an excellent example of d/acc principles: they allow users and communities to verify trustworthiness without compromising privacy, and protect their security without relying on centralized choke points that impose their own definitions of who is good and bad. they improve global accessibility by creating better and fairer ways to protect a user or service's security than common techniques used today, such as discriminating against entire countries that are deemed untrustworthy. these are very powerful primitives that could be necessary if we want to preserve a decentralized vision of information security going into the 21st century. working on defensive technologies for cyberspace more broadly can make the internet more open, safe and free in very important ways going forward. info-defense cyber-defense, as i have described it, is about situations where it's easy for reasonable human beings to all come to consensus on who the attacker is. if someone tries to hack into your wallet, it's easy to agree that the hacker is the bad guy. if someone tries to dos attack a website, it's easy to agree that they're being malicious, and are not morally the same as a regular user trying to read what's on the site. there are other situations where the lines are more blurry. it is the tools for improving our defense in these situations that i call "info-defense". take, for example, fact checking (aka, preventing "misinformation"). i am a huge fan of community notes, which has done a lot to help users identify truths and falsehoods in what other users are tweeting. community notes uses a new algorithm which surfaces not the notes that are the most popular, but rather the notes that are most approved by users across the political spectrum. community notes in action. i am also a fan of prediction markets, which can help identify the significance of events in real time, before the dust settles and there is consensus on which direction is which. the polymarket on sam altman is very helpful in giving a useful summary of the ultimate consequences of hour-by-hour revelations and negotiations, giving much-needed context to people who only see the individual news items and don't understand the significance of each one. prediction markets are often flawed. but twitter influencers who are willing to confidently express what they think "will" happen over the next year are often even more flawed. there is still room to improve prediction markets much further. for example, a major practical flaw of prediction markets is their low volume on all but the most high-profile events; a natural direction to try to solve this would be to have prediction markets that are played by ais. within the blockchain space, there is a particular type of info defense that i think we need much more of. namely, wallets should be much more opinionated and active in helping users determine the meaning of things that they are signing, and protecting them from fraud and scams. this is an intermediate case: what is and is not a scam is less subjective than perspectives on controversial social events, but it's more subjective than telling apart legitimate users from dos attackers or hackers. metamask has an scam database already, and automatically blocks users from visiting scam sites: applications like fire are an example of one way to go much further. however, security software like this should not be something that requires explicit installs; it should be part of crypto wallets, or even browsers, by default. because of its more subjective nature, info-defense is inherently more collective than cyber-defense: you need to somehow plug into a large and sophisticated group of people to identify what might be true or false, and what kind of application is a deceptive ponzi. there is an opportunity for developers to go much further in developing effective info-defense, and in hardening existing forms of info-defense. something like community notes could be included in browsers, and cover not just social media platforms but also the whole internet. social technology beyond the "defense" framing to some degree, i can be justifiably accused of shoehorning by describing some of these info technologies as being about "defense". after all, defense is about helping well-meaning actors be protected from badly-intentioned actors (or, in some cases, from nature). some of these social technologies, however, are about helping well-intentioned actors form consensus. a good example of this is pol.is, which uses an algorithm similar to community notes (and which predates community notes) to help communities identify points of agreement between sub-tribes who otherwise disagree on a lot. viewpoints.xyz was inspired by pol.is, and has a similar spirit: technologies like this could be used to enable more decentralized governance over contentious decisions. again, blockchain communities are a good testing ground for this, and one where such algorithms have already shown valuable. generally, decisions over which improvements ("eips") to make to the ethereum protocol are made by a fairly small group in meetings called "all core devs calls". for highly technical decisions, where most community members have no strong feelings, this works reasonably well. for more consequential decisions, which affect protocol economics, or more fundamental values like immutability and censorship resistance, this is often not enough. back in 2016-17, when a series of contentious decisions around implementing the dao fork, reducing issuance and (not) unfreezing the parity wallet, tools like carbonvote, as well as social media voting, helped the community and the developers to see which way the bulk of the community opinion was facing. carbonvote on the dao fork. carbonvote had its flaws: it relied on eth holdings to determine who was a member of the ethereum community, making the outcome dominated by a few wealthy eth holders ("whales"). with modern tools, however, we could make a much better carbonvote, leveraging multiple signals such as poaps, zupass stamps, gitcoin passports, protocol guild memberships, as well as eth (or even solo-staked-eth) holdings to gauge community membership. tools like this could be used by any community to make higher-quality decisions, find points of commonality, coordinate (physical or digital) migrations or do a number of other things without relying on opaque centralized leadership. this is not defense acceleration per se, but it can certainly be called democracy acceleration. such tools could even be used to improve and democratize the governance of key actors and institutions working in ai. so what are the paths forward for superintelligence? the above is all well and good, and could make the world a much more harmonious, safer and freer place for the next century. however, it does not yet address the big elephant in the room: superintelligent ai. the default path forward suggested by many of those who worry about ai essentially leads to a minimal ai world government. near-term versions of this include a proposal for a "multinational agi consortium" ("magic"). such a consortium, if it gets established and succeeds at its goals of creating superintelligent ai, would have a natural path to becoming a de-facto minimal world government. longer-term, there are ideas like the "pivotal act" theory: we create an ai that performs a single one-time act which rearranges the world into a game where from that point forward humans are still in charge, but where the game board is somehow more defense-favoring and more fit for human flourishing. the main practical issue that i see with this so far is that people don't seem to actually trust any specific governance mechanism with the power to build such a thing. this fact becomes stark when you look at the results to my recent twitter polls, asking if people would prefer to see ai monopolized by a single entity with a decade head-start, or ai delayed by a decade for everyone: the size of each poll is small, but the polls make up for it in the uniformity of their result across a wide diversity of sources and options. in nine out of nine cases, the majority of people would rather see highly advanced ai delayed by a decade outright than be monopolized by a single group, whether it's a corporation, government or multinational body. in seven out of nine cases, delay won by at least two to one. this seems like an important fact to understand for anyone pursuing ai regulation. current approaches have been focusing on creating licensing schemes and regulatory requirements, trying to restrict ai development to a smaller number of people, but these have seen popular pushback precisely because people don't want to see anyone monopolize something so powerful. even if such top-down regulatory proposals reduce risks of extinction, they risk increasing the chance of some kind of permanent lock-in to centralized totalitarianism. paradoxically, could agreements banning extremely advanced ai research outright (perhaps with exceptions for biomedical ai), combined with measures like mandating open source for those models that are not banned as a way of reducing profit motives while further improving equality of access, be more popular? the main approach preferred by opponents of the "let's get one global org to do ai and make its governance really really good" route is polytheistic ai: intentionally try to make sure there's lots of people and companies developing lots of ais, so that none of them grows far more powerful than the other. this way, the theory goes, even as ais become superintelligent, we can retain a balance of power. this philosophy is interesting, but my experience trying to ensure "polytheism" within the ethereum ecosystem does make me worry that this is an inherently unstable equilibrium. in ethereum, we have intentionally tried to ensure decentralization of many parts of the stack: ensuring that there's no single codebase that controls more than half of the proof of stake network, trying to counteract the dominance of large staking pools, improving geographic decentralization, and so on. essentially, ethereum is actually attempting to execute on the old libertarian dream of a market-based society that uses social pressure, rather than government, as the antitrust regulator. to some extent, this has worked: the prysm client's dominance has dropped from above 70% to under 45%. but this is not some automatic market process: it's the result of human intention and coordinated action. my experience within ethereum is mirrored by learnings from the broader world as a whole, where many markets have proven to be natural monopolies. with superintelligent ais acting independently of humans, the situation is even more unstable. thanks to recursive self-improvement, the strongest ai may pull ahead very quickly, and once ais are more powerful than humans, there is no force that can push things back into balance. additionally, even if we do get a polytheistic world of superintelligent ais that ends up stable, we still have the other problem: that we get a universe where humans are pets. a happy path: merge with the ais? a different option that i have heard about more recently is to focus less on ai as something separate from humans, and more on tools that enhance human cognition rather than replacing it. one near-term example of something that goes in this direction is ai drawing tools. today, the most prominent tools for making ai-generated images only have one step at which the human gives their input, and ai fully takes over from there. an alternative would be to focus more on ai versions of photoshop: tools where the artist or the ai might make an early draft of a picture, and then the two collaborate on improving it with a process of real-time feedback. photoshop generative ai fill, 2023. source. i tried, it and it takes time to get used to but it actually works quite well! another direction in a similar spirit is the open agency architecture, which proposes splitting the different parts of an ai "mind" (eg. making plans, executing on plans, interpreting information from the outside world) into separate components, and introducing diverse human feedback in between these parts. so far, this sounds mundane, and something that almost everyone can agree that it would be good to have. the economist daron acemoglu's work is far from this kind of ai futurism, but his new book power and progress hints at wanting to see more of exactly these types of ai. but if we want to extrapolate this idea of human-ai cooperation further, we get to more radical conclusions. unless we create a world government powerful enough to detect and stop every small group of people hacking on individual gpus with laptops, someone is going to create a superintelligent ai eventually one that can think a thousand times faster than we can and no combination of humans using tools with their hands is going to be able to hold its own against that. and so we need to take this idea of human-computer cooperation much deeper and further. a first natural step is brain-computer interfaces. brain-computer interfaces can give humans much more direct access to more-and-more powerful forms of computation and cognition, reducing the two-way communication loop between man and machine from seconds to milliseconds. this would also greatly reduce the "mental effort" cost to getting a computer to help you gather facts, give suggestions or execute on a plan. later stages of such a roadmap admittedly get weird. in addition to brain-computer interfaces, there are various paths to improving our brains directly through innovations in biology. an eventual further step, which merges both paths, may involve uploading our minds to run on computers directly. this would also be the ultimate d/acc for physical security: protecting ourselves from harm would no longer be a challenging problem of protecting inevitably-squishy human bodies, but rather a much simpler problem of making data backups. directions like this are sometimes met with worry, in part because they are irreversible, and in part because they may give powerful people more advantages over the rest of us. brain-computer interfaces in particular have dangers after all, we are talking about literally reading and writing to people's minds. these concerns are exactly why i think it would be ideal for a leading role in this path to be held by a security-focused open-source movement, rather than closed and proprietary corporations and venture capital funds. additionally, all of these issues are worse with superintelligent ais that operate independently from humans, than they are with augmentations that are closely tied to humans. the divide between "enhanced" and "unenhanced" already exists today due to limitations in who can and can't use chatgpt. if we want a future that is both superintelligent and "human", one where human beings are not just pets, but actually retain meaningful agency over the world, then it feels like something like this is the most natural option. there are also good arguments why this could be a safer ai alignment path: by involving human feedback at each step of decision-making, we reduce the incentive to offload high-level planning responsibility to the ai itself, and thereby reduce the chance that the ai does something totally unaligned with humanity's values on its own. one other argument in favor of this direction is that it may be more socially palatable than simply shouting "pause ai" without a complementary message providing an alternative path forward. it will require a philosophical shift from the current mentality that tech advancements that touch humans are dangerous but advancements that are separate from humans are by-default safe. but it has a huge countervailing benefit: it gives developers something to do. today, the ai safety movement's primary message to ai developers seems to be "you should just stop". one can work on alignment research, but today this lacks economic incentives. compared to this, the common e/acc message of "you're already a hero just the way you are" is understandably extremely appealing. a d/acc message, one that says "you should build, and build profitable things, but be much more selective and intentional in making sure you are building things that help you and humanity thrive", may be a winner. is d/acc compatible with your existing philosophy? if you are an e/acc, then d/acc is a subspecies of e/acc just one that is much more selective and intentional. if you are an effective altruist, then d/acc is a re-branding of the effective-altruist idea of differential technology development, though with a greater emphasis on liberal and democratic values. if you are a libertarian, then d/acc is a sub-species of techno-libertarianism, though a more pragmatic one that is more critical of "the techno-capital machine", and willing to accept government interventions today (at least, if cultural interventions don't work) to prevent much worse un-freedom tomorrow. if you are a pluralist, in the glen weyl sense of the term, then d/acc is a frame that can easily include the emphasis on better democratic coordination technology that plurality values. if you are a public health advocate, then d/acc ideas can be a source of a broader long-term vision, and opportunity to find common ground with "tech people" that you might otherwise feel at odds with. if you are a blockchain advocate, then d/acc is a more modern and broader narrative to embrace than the fifteen-year-old emphasis on hyperinflation and banks, which puts blockchains into context as one of many tools in a concrete strategy to build toward a brighter future. if you are a solarpunk, then d/acc is a subspecies of solarpunk, and incorporates a similar emphasis on intentionality and collective action. if you are a lunarpunk, then you will appreciate the d/acc emphasis on informational defense, through maintaining privacy and freedom. we are the brightest star i love technology because technology expands human potential. ten thousand years ago, we could build some hand tools, change which plants grow on a small patch of land, and build basic houses. today, we can build 800-meter-tall towers, store the entirety of recorded human knowledge in a device we can hold in our hands, communicate instantly across the globe, double our lifespan, and live happy and fulfilling lives without fear of our best friends regularly dropping dead of disease. we started from the bottom, now we're here. i believe that these things are deeply good, and that expanding humanity's reach even further to the planets and stars is deeply good, because i believe humanity is deeply good. it is fashionable in some circles to be skeptical of this: the voluntary human extinction movement argues that the earth would be better off without humans existing at all, and many more want to see much smaller number of human beings see the light of this world in the centuries to come. it is common to argue that humans are bad because we cheat and steal, engage in colonialism and war, and mistreat and annihilate other species. my reply to this style of thinking is one simple question: compared to what? yes, human beings are often mean, but we much more often show kindness and mercy, and work together for our common benefit. even during wars we often take care to protect civilians certainly not nearly enough, but also far more than we did 2000 years ago. the next century may well bring widely available non-animal-based meat, eliminating the largest moral catastrophe that human beings can justly be blamed for today. non-human animals are not like this. there is no situation where a cat will adopt an entire lifestyle of refusing to eat mice as a matter of ethical principle. the sun is growing brighter every year, and in about one billion years, it is expected that this will make the earth too hot to sustain life. does the sun even think about the genocide that it is going to cause? and so it is my firm belief that, out of all the things that we have known and seen in our universe, we, humans, are the brightest star. we are the one thing that we know about that, even if imperfectly, sometimes make an earnest effort to care about "the good", and adjust our behavior to better serve it. two billion years from now, if the earth or any part of the universe still bears the beauty of earthly life, it will be human artifices like space travel and geoengineering that will have made it happen. we need to build, and accelerate. but there is a very real question that needs to be asked: what is the thing that we are accelerating towards? the 21st century may well be the pivotal century for humanity, the century in which our fate for millennia to come gets decided. do we fall into one of a number of traps from which we cannot escape, or do we find a way toward a future where we retain our freedom and agency? these are challenging problems. but i look forward to watching and participating in our species' grand collective effort to find the answers. nick szabo confirmed as keynote speaker of ethereum’s devcon1 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search nick szabo confirmed as keynote speaker of ethereum’s devcon1 posted by george hallam on october 22, 2015 events london, united kingdom, october 22nd the ethereum foundation is pleased to announce the keynote speaker of ethereum’s annual conference dξvcon1, nick szabo. nick’s keynote will focus on the history of the blockchain, smart contracts and assets on the blockchain. nick is a computer scientist, legal scholar and cryptographer known for his research on digital contracts and digital currency. the phrase and concept of “smart contracts” was developed by szabo with the intent of merging contract law into programming protocols on the internet. recently, szabo contrasted the use of smart contracts on ethereum in relation to the bitcoin blockchain, “if you want to have a flexible general purpose programming environment like programmers have been used to since the 1950s at least, then you’re going to want to use ethereum because it’s got a turing complete language and a large state. i sometimes make the comparison of a pocket calculator [bitcoin] versus, say, a general purpose computer [ethereum].” ethereum founder vitalik buterin noted, “we are thrilled to welcome nick to dξvcon1. his pioneering work on smart contracts and the evolution of blockchain technology will add a tremendously valuable perspective to the conference”. dξvcon1 will be held from november 9th-13th at gibson hall in london, uk. it will feature five days of technical presentations and events on research and core protocols, decentralised application (dapp) development, and industry and social implications. ethereum is a decentralized blockchain protocol that executes smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference. ethereum is how the internet was supposed to work.  ### previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle why we need wide adoption of social recovery wallets 2021 jan 11 see all posts special thanks to itamar lesuisse from argent and daniel wang from loopring for feedback. one of the great challenges with making cryptocurrency and blockchain applications usable for average users is security: how do we prevent users' funds from being lost or stolen? losses and thefts are a serious issue, often costing innocent blockchain users thousands of dollars or even in some cases the majority of their entire net worth. there have been many solutions proposed over the years: paper wallets, hardware wallets, and my own one-time favorite: multisig wallets. and indeed they have led to significant improvements in security. however, these solutions have all suffered from various defects sometimes providing far less extra protection against theft and loss than is actually needed, sometimes being cumbersome and difficult to use leading to very low adoption, and sometimes both. but recently, there is an emerging better alternative: a newer type of smart contract wallet called a social recovery wallet. these wallets can potentially provide a high level of security and much better usability than previous options, but there is still a way to go before they can be easily and widely deployed. this post will go through what social recovery wallets are, why they matter, and how we can and should move toward much broader adoption of them throughout the ecosystem. wallet security is a really big problem wallet security issues have been a thorn in the side of the blockchain ecosystem almost since the beginning. cryptocurrency losses and thefts were rampant even back in 2011 when bitcoin was almost the only cryptocurrency out there; indeed, in my pre-ethereum role as a cofounder and writer of bitcoin magazine, i wrote an entire article detailing the horrors of hacks and losses and thefts that were already happening at the time. here is one sample: last night around 9pm pdt, i clicked a link to go to coinchat[.]freetzi[.]com – and i was prompted to run java. i did (thinking this was a legitimate chatoom), and nothing happened. i closed the window and thought nothing of it. i opened my bitcoin-qt wallet approx 14 minutes later, and saw a transaction that i did not approve go to wallet 1es3qvvkn1qa2p6me7jlcvmzpqxvxwpntc for almost my entire wallet... this person's losses were 2.07 btc, worth $300 at the time, and over $70000 today. here's another one: in june 2011, the bitcointalk member "allinvain" lost 25,000 btc (worth $500,000 at the time) after an unknown intruder somehow gained direct access to his computer. the attacker was able to access allinvain's wallet.dat file, and quickly empty out the wallet – either by sending a transaction from allinvain's computer itself, or by simply uploading the wallet.dat file and emptying it on his own machine. in present-day value, that's a loss of nearly one billion dollars. but theft is not the only concern; there are also losses from losing one's private keys. here's stefan thomas: bitcoin developer stefan thomas had three backups of his wallet – an encrypted usb stick, a dropbox account and a virtualbox virtual machine. however, he managed to erase two of them and forget the password to the third, forever losing access to 7,000 btc (worth $125,000 at the time). thomas's reaction: "[i'm] pretty dedicated to creating better clients since then." one analysis of the bitcoin ecosystem suggests that 1500 btc may be lost every day over ten times more than what bitcoin users spend on transaction fees, and over the years adding up to as much as 20% of the total supply. the stories and the numbers alike point to the same inescapable truth: the importance of the wallet security problem is great, and it should not be underestimated. it's easy to see the social and psychological reasons why wallet security is easy to underestimate: people naturally worry about appearing uncareful or dumb in front of an always judgemental public, and so many keep their experiences with their funds getting hacked to themselves. loss of funds is even worse, as there is a pervasive (though in my opinion very incorrect) feeling that "there is no one to blame but yourself". but the reality is that the whole point of digital technology, blockchains included, is to make it easier for humans to engage in very complicated tasks without having to exert extreme mental effort or live in constant fear of making mistakes. an ecosystem whose only answer to losses and thefts is a combination of 12-step tutorials, not-very-secure half-measures and the not-so-occasional semi-sarcastic "sorry for your loss" is going to have a hard time getting broad adoption. so solutions that reduce the quantity of losses and thefts taking place, without requiring all cryptocurrency users to turn personal security into a full-time hobby, are highly valuable for the industry. hardware wallets alone are not good enough hardware wallets are often touted as the best-in-class technology for cryptocurrency funds management. a hardware wallet is a specialized hardware device which can be connected to your computer or phone (eg. through usb), and which contains a specialized chip that can only generate private keys and sign transactions. a transaction would be initiated on your computer or phone, must be confirmed on the hardware wallet before it can be sent. the private key stays on your hardware wallet, so an attacker that hacks into your computer or phone could not drain the funds. hardware wallets are a significant improvement, and they certainly would have protected the java chatroom victim, but they are not perfect. i see two main problems with hardware wallets: supply chain attacks: if you buy a hardware wallet, you are trusting a number of actors that were involved in producing it the company that designed the wallet, the factory that produced it, and everyone involved in shipping it who could have replaced it with a fake. hardware wallets are potentially a magnet for such attacks: the ratio of funds stolen to number of devices compromised is very high. to their credit, hardware wallet manufacturers such as ledger have put in many safeguards to protect against these risks, but some risks still remain. a hardware device fundamentally cannot be audited the same way a piece of open source software can. still a single point of failure: if someone steals your hardware wallet right after they stand behind your shoulder and catch you typing in the pin, they can steal your funds. if you lose your hardware wallet, then you lose your funds unless the hardware wallet generates and outputs a backup at setup time, but as we will see those have problems of their own... mnemonic phrases are not good enough many wallets, hardware and software alike, have a setup procedure during which they output a mnemonic phrase, which is a human-readable 12 to 24-word encoding of the wallet's root private key. a mnemonic phrase looks like this: vote dance type subject valley fall usage silk essay lunch endorse lunar obvious race ribbon key already arrow enable drama keen survey lesson cruel if you lose your wallet but you have the mnemonic phrase, you can input the phrase when setting up a new wallet to recover your account, as the mnemonic phrase contains the root key from which all of your other keys can be generated. mnemonic phrases are good for protecting against loss, but they do nothing against theft. even worse, they add a new vector for theft: if you have the standard hardware wallet + mnemonic backup combo, then someone stealing either your hardware wallet + pin or your mnemonic backup can steal your funds. furthermore, maintaining a mnemonic phrase and not accidentally throwing it away is itself a non-trivial mental effort. the problems with theft can be alleviated if you split the phrase in half and give half to your friend, but (i) almost no one actually promotes this, (ii) there are security issues, as if the phrase is short (128 bits) then a sophisticated and motivated attacker who steals one piece may be able to brute-force through all \(2^{64}\) possible combinations to find the other, and (iii) it increases the mental overhead even further. so what do we need? what we need is a wallet design which satisfies three key criteria: no single point of failure: there is no single thing (and ideally, no collection of things which travel together) which, if stolen, can give an attacker access to your funds, or if lost, can deny you access to your funds. low mental overhead: as much as possible, it should not require users to learn strange new habits or exert mental effort to always remember to follow certain patterns of behavior. maximum ease of transacting: most normal activities should not require much more effort than they do in regular wallets (eg. status, metamask...) multisig is good! the best-in-class technology for solving these problems back in 2013 was multisig. you could have a wallet that has three keys, where any two of them are needed to send a transaction. this technology was originally developed within the bitcoin ecosystem, but excellent multisig wallets (eg. see gnosis safe) now exist for ethereum too. multisig wallets have been highly successful within organizations: the ethereum foundation uses a 4-of-7 multisig wallet to store its funds, as do many other orgs in the ethereum ecosystem. for a multisig wallet to hold the funds for an individual, the main challenge is: who holds the funds, and how are transactions approved? the most common formula is some variant of "two easily accessible, but separate, keys, held by you (eg. laptop and phone) and a third more secure but less accessible a backup, held offline or by a friend or institution". this is reasonably secure: there is no single device that can be lost or stolen that would lead to you losing access to your funds. but the security is far from perfect: if you can steal someone's laptop, it's often not that hard to steal their phone as well. the usability is also a challenge, as every transaction now requires two confirmations with two devices. social recovery is better this gets us to my preferred method for securing a wallet: social recovery. a social recovery system works as follows: there is a single "signing key" that can be used to approve transactions there is a set of at least 3 (or a much higher number) of "guardians", of which a majority can cooperate to change the signing key of the account. the signing key has the ability to add or remove guardians, though only after a delay (often 1-3 days). under all normal circumstances, the user can simply use their social recovery wallet like a regular wallet, signing messages with their signing key so that each transaction signed can fly off with a single confirmation click much like it would in a "traditional" wallet like metamask. if a user loses their signing key, that is when the social recovery functionality would kick in. the user can simply reach out to their guardians and ask them to sign a special transaction to change the signing pubkey registered in the wallet contract to a new one. this is easy: they can simply go to a webpage such as security.loopring.io, sign in, see a recovery request and sign it. about as easy for each guardian as making a uniswap trade. there are many possible choices for whom to select as a guardian. the three most common choices are: other devices (or paper mnemonics) owned by the wallet holder themselves friends and family members institutions, which would sign a recovery message if they get a confirmation of your phone number or email or perhaps in high value cases verify you personally by video call guardians are easy to add: you can add a guardian simply by typing in their ens name or eth address, though most social recovery wallets will require the guardian to sign a transaction in the recovery webpage to agree to be added. in any sanely designed social recovery wallet, the guardian does not need to download and use the same wallet; they can simply use their existing ethereum wallet, whichever type of wallet it is. given the high convenience of adding guardians, if you are lucky enough that your social circles are already made up of ethereum users, i personally favor high guardian counts (ideally 7+) for increased security. if you already have a wallet, there is no ongoing mental effort required to be a guardian: any recovery operations that you do would be done through your existing wallet. if you not know many other active ethereum users, then a smaller number of guardians that you trust to be technically competent is best. to reduce the risk of attacks on guardians and collusion, your guardians do not have to be publicly known: in fact, they do not need to know each other's identities. this can be accomplished in two ways. first, instead of the guardians' addresses being stored directly on chain, a hash of the list of addresses can be stored on chain, and the wallet owner would only need to publish the full list at recovery time. second, each guardian can be asked to deterministically generate a new single-purpose address that they would use just for that particular recovery; they would not need to actually send any transactions with that address unless a recovery is actually required. to complement these technical protections, it's recommended to choose a diverse collection of guardians from different social circles (including ideally one institutional guardian); these recommendations together would make it extremely difficult for the guardians to be attacked simultaneously or collude. in the event that you die or are permanently incapacitated, it would be a socially agreed standard protocol that guardians can publicly announce themselves, so in that case they can find each other and recover your funds. social recovery wallets are not a betrayal, but rather an expression, of "crypto values" one common response to suggestions to use any form of multisig, social recovery or otherwise, is the idea that this solution goes back to "trusting people", and so is a betrayal of the values of the blockchain and cryptocurrency industry. while i understand why one may think this at first glance, i would argue that this criticism stems from a fundamental misunderstanding of what crypto should be about. to me, the goal of crypto was never to remove the need for all trust. rather, the goal of crypto is to give people access to cryptographic and economic building blocks that give people more choice in whom to trust, and furthermore allow people to build more constrained forms of trust: giving someone the power to do some things on your behalf without giving them the power to do everything. viewed in this way, multisig and social recovery are a perfect expression of this principle: each participant has some influence over the ability to accept or reject transactions, but no one can move funds unilaterally. this more complex logic allows for a setup far more secure than what would be possible if there had to be one person or key that unilaterally controlled the funds. this fundamental idea, that human inputs should be wielded carefully but not thrown away outright, is powerful because it works well with the strengths and weaknesses of the human brain. the human brain is quite poorly suited for remembering passwords and tracking paper wallets, but it's an asic for keeping track of relationships with other people. this effect is even stronger for less technical users: they may have a harder time with wallets and passwords, but they are just as adept at social tasks like "choose 7 people who won't all gang up on me". if we can extract at least some information from human inputs into a mechanism, without those inputs turning into a vector for attack and exploitation, then we should figure out how. and social recovery is very robust: for a wallet with 7 guardians to be compromised, 4 of the 7 guardians would need to somehow discover each other and agree to steal the funds, without any of them tipping the owner off: certainly a much tougher challenge than attacking a wallet protected purely by a single individuals. how can social recovery protect against theft? social recovery as explained above deals with the risk that you lose your wallet. but there is still the risk that your signing key gets stolen: someone hacks into your computer, sneaks up behind you while you're already logged in and hits you over the head, or even just uses some user interface glitch to trick you into signing a transaction that you did not intend to sign. we can extend social recovery to deal with such issues by adding a vault. every social recovery wallet can come with an automatically generated vault. assets can be moved to the vault just by sending them to the vault's address, but they can be moved out of the vault only with a 1 week delay. during that delay, the signing key (or, by extension, the guardians) can cancel the transaction. if desired, the vault could also be programmed so that some limited financial operations (eg. uniswap trades between some whitelisted tokens) can be done without delay. existing social recovery wallets currently, the two major wallets that have implemented social recovery are the argent wallet and the loopring wallet: the argent wallet is the first major, and still the most popular, "smart contract wallet" currently in use, and social recovery is one of its main selling points. the argent wallet includes an interface by which guardians can be added and removed: to protect against theft, the wallet has a daily limit: transactions up to that amount are instant but transactions above that amount require guardians to approve to finalize the withdrawal. the loopring wallet is most known for being built by the developers of (and of course including support for) the loopring protocol, a zk rollup for payments and decentralized exchange. but the loopring wallet also has a social recovery feature, which works very similarly to that in argent. in both cases, the wallet companies provide one guardian for free, which relies on a confirmation code sent by mobile phone to authenticate you. for the other guardians, you can add either other users of the same wallet, or any ethereum user by providing their ethereum address. the user experience in both cases is surprisingly smooth. there were two main challenges. first, the smoothness in both cases relies on a central "relayer" run by the wallet maker that re-publishes signed messages as transactions. second, the fees are high. fortunately, both of these problems are surmountable. migration to layer 2 (rollups) can solve the remaining challenges as mentioned above, there are two key challenges: (i) the dependence on relayers to solve transactions, and (ii) high transaction fees. the first challenge, dependence on relayers, is an increasingly common problem in ethereum applications. the issue arises because there are two types of accounts in ethereum: externally owned accounts (eoas), which are accounts controlled by a single private key, and contracts. in ethereum, there is a rule that every transaction must start from an eoa; the original intention was that eoas represent "users" and contracts represent "applications", and an application can only run if a user talks to the application. if we want wallets with more complex policies, like multisig and social recovery, we need to use contracts to represent users. but this poses a challenge: if your funds are in a contract, you need to have some other account that has eth that can pay to start each transaction, and it needs quite a lot of eth just in case transaction fees get really high. argent and loopring get around this problem by personally running a "relayer". the relayer listens for off-chain digitally signed "messages" submitted by users, and wraps these messages in a transaction and publishes them to chain. but for the long run, this is a poor solution; it adds an extra point of centralization. if the relayer is down and a user really needs to send a transaction, they can always just send it from their own eoa, but it is nevertheless the case that a new tradeoff between centralization and inconvenience is introduced. there are efforts to solve this problem and get convenience without centralization; the main two categories revolve around either making a generalized decentralized relayer network or modifying the ethereum protocol itself to allow transactions to begin from contracts. but neither of these solutions solve transaction fees, and in fact, they make the problem worse due to smart contracts' inherently greater complexity. fortunately, we can solve both of these problems at the same time, by looking toward a third solution: moving the ecosystem onto layer 2 protocols such as optimistic rollups and zk rollups. optimistic and zk rollups can both be designed with account abstraction built in, circumventing any need for relayers. existing wallet developers are already looking into rollups, but ultimately migrating to rollups en masse is an ecosystem-wide challenge. an ecosystem-wide mass migration to rollups is as good an opportunity as any to reverse the ethereum ecosystem's earlier mistakes and give multisig and smart contract wallets a much more central role in helping to secure users' funds. but this requires broader recognition that wallet security is a challenge, and that we have not gone nearly as far in trying to meet and challenge as we should. multisig and social recovery need not be the end of the story; there may well be designs that work even better. but the simple reform of moving to rollups and making sure that these rollups treat smart contract wallets as first class citizens is an important step toward making that happen. announcing the new foundation board and executive director | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search announcing the new foundation board and executive director posted by stephan tual on july 30, 2015 organizational after an intensive recruitment period, the ethereum foundation has selected an executive director and named three members to the board of directors. this is a new organizational structure, which will allow us to focus on our next phase of development. the ethereum foundation is a not for profit (‘stiftung’) organization registered in switzerland, and has the purpose of managing the funds that were raised from the ether sale in order to best serve the ethereum and decentralized technology ecosystem. our new executive director is ming chan, and our board is currently comprised of four members. they are: lars klawitter, vadim levitin, wayne hennessy-barrett and vitalik buterin (president of the board). executive director ming chan an alumni of massachusetts institute of technology, where she studied computer science, media arts & sciences, ming is a swiss-born chinese american who has spent decades leading and completing complex it and management consulting projects, founding and growing business ventures, and working with top educators, scientists, and inventors to bring inspiring research innovation to life. she has followed ethereum since 2013 and has a sound understanding of the legal and regulatory issues relevant to blockchain technology. linkedin, twitter board members lars klawitter lars has a degree in computer science and was involved with startups as a founder or senior executive during the first internet revolution of the late 90ies. having since moved into senior positions in the automotive industry, lars has substantial experience in building and growing organisations at the intersection of technology and innovation. he held several c level it positions, the most recent of which cio at rolls-royce motor cars, the company where he is currently responsible for the special vehicles and innovations business. linkedin, twitter wayne hennessy-barrett founder and ceo of 4g capital, an african mobile money fintech business based in kenya. he has created and led scalable distribution systems in emerging markets around the world with extensive experience in corporate governance and risk management. after a distinguished military career he has provided intelligence-led consultancy and strategic advice to a number of boards in both the commercial and public sectors. he is a committed advocate for the transformative power of financial technology for the alleviation of poverty and sustainable development. linkedin dr. vadim david levitin vadim worked on technology issues in twenty-three countries on behalf of the united nations. trained as a physician, he found the nexus between human and technology systems intriguing and illustrative leading him to pursue a master’s degree in technology. he ran a business development company in eastern europe serving fortune 500 clients interested in those markets as the region emerged from communism. later, he was ceo of ecommerce institute, a global technology company with subsidiaries in the u.k., singapore, malaysia, australia, new zealand, and the caribbean. he has consulted with major companies, governments, and ngos on almost every continent, authored national strategies on economic and technology issues ranging from digital marketplaces to online stock exchanges, from e-schools to technology strategies for emerging countries, from transportation to tourism, and from healthcare to national infrastructure broadband development. linkedin vitalik buterin vitalik is one of the inventors of ethereum. vitalik first discovered bitcoin in 2011, and the combination of math, economics and an immediate feeling that it represented the way that money "should be" attracted him immediately to the scene. in september 2011 he became the cofounder and original head writer for bitcoin magazine. following this, vitalik founded ethereum, a blockchain-based arbitrary-state turing-complete scripting platform useful for smart contracts, computational resource marketplaces, financial experimentation, decentralized governance and many other use cases that we have not even thought of yet. linkedin, twitter if you would like to organize an interview with our new foundation board, please contact press@ethereum.org previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the evolution of ethereum | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the evolution of ethereum posted by vitalik buterin on september 28, 2015 research & development many of you know that the ethereum platform grew out of the realization that blockchains can go far beyond currency, together with a frustration with the limitations of previous projects. the core idea was simple: a blockchain with a built-in turing-complete programming language, allowing users to build any kind of applications on top. over time, the vision evolved and expanded. the blockchain remains a crucial centerpiece, but it is ultimately only part of a larger vision of “web 3.0” as described by gavin wood here: a more secure, trustworthy and globally accessible internet for agreements, finance, auditing, tracking and simple websites and web applications that use decentralized technology to overcome some of the practical, political and technological inefficiencies of previous approaches. together with the ethereum blockchain, we see an entire suite of internally and externally developed, low-level and high-level protocols including solidity, whisper, ipfs, zero knowledge proof systems, account management systems, dedicated browsers and much more, all with the goal of providing a coherent vision of the internet as it should be. with such an ambitious vision, come some challenges. right now, the ethereum project is in a time of complicated transition. many of the difficult initial work myself, gavin, jeff, martin, lefteris, felix, vlad and many others developing four compatible versions of a project that our security auditors have described as having “testing needs … more complex than anything [they’ve] looked at before”, christoph and dmitry’s tireless efforts setting up over ten thousand tests, marian, taylor and konstantin's work on network analysis and emergency response architecture, christian, liana and gavin’s work on getting solidity off the ground, imapp's work on the jit evm, and the many other projects of contributors to the ethereum platform  of which there are too many to mention, all culminating with the successful launch of a blockchain with over 50millionworthofvaluefloatingaroundonitstartingfromdayone,arenowbehindus.ethereumnowactuallyexists,andthosewhowereluckyenoughtotakeup<ahref="http://www.contravex.com/2014/07/23/a−guide−to−buying−5000−ether−bitcoin−2−5x−more−than−ethereums−genesis−sale/">mirceapopescu′soffer</a>tobuyethfromhimthathedoesn′thaveatapriceof50 million worth of value floating around on it starting from day one, are now behind us. ethereum now actually exists, and those who were lucky enough to take up <a href="http://www.contravex.com/2014/07/23/a-guide-to-buying-5000-ether-bitcoin-2-5x-more-than-ethereums-genesis-sale/">mircea popescu's offer</a> to buy eth from him that he doesn't have at a price of 50millionworthofvaluefloatingaroundonitstartingfromdayone,arenowbehindus.ethereumnowactuallyexists,andthosewhowereluckyenoughtotakeup<ahref="http://www.contravex.com/2014/07/23/a−guide−to−buying−5000−ether−bitcoin−2−5x−more−than−ethereums−genesis−sale/">mirceapopescu′soffer</a>tobuyethfromhimthathedoesn′thaveatapriceof0.12 a piece are welcome to try their best to collect. today, we can all be proud that the ethereum developer ecosystem has grown large enough to include major banks, corporations, governments, over a hundred dapps and individuals and businesses in dozens of countries speaking dozens of languages. at the same time, however, there are some difficult challenges that remain: some technical, some organizational, and some of almost every kind. the core of the problem is simple. up until fairly recently, almost all of the work that has been done on the ethereum project has been done by subsidiaries of the foundation. in the future, however, although the foundation and its subsidiaries are going to continue to take on a strong and leading role, it will be the community that will gradually be the primary driver in making it succeed. this is true for several reasons, some unplanned and some positive. first of all, it is indeed true that the foundation’s finances are limited, and a large part of this was the result of our failure to sell nearly as much of our btc holdings as we were planning to before the price dropped to 220;asaresult,wesufferedroughly220; as a result, we suffered roughly 220;asaresult,wesufferedroughly9m in lost potential capital , and a hiring schedule that was meant to last over three years ended up lasting a little under two (although bolstered by a “second wind” from our eth holdings). second, the project’s needs have grown. over the past twenty months, the project has grown from being a simple attempt to improve on mastercoin by adding a programming language into an effort to push forward a powerful and expansive vision of "web 3.0" that includes multiple technologies, some built by ourselves and some by others, and a complex software stack that integrates them all with one simple aim: to make it as easy to build secure, globally accessible and trust-minimized decentralized applications as it is to build a website and hopefully even easier. the foundation and its subsidiaries alone simply do not have the manpower to push the entirety of this vision through to its ultimate completion, including proof-of-stake driven scalable blockchains, seamlessly integrated distributed hash tables, programming languages with formal verification systems backed by state-of-the-art theorem provers and dozens of categories of middleware, all by itself; although the foundation and its subsidiaries can, and will, continue to be the primary driver of technology at the core, a highly community-driven model is necessary and essential, both to help the ethereum ecosystem maximally grow and flourish and to establish ethereum as a decentralized project which is ultimately owned by all of humanity, and not any one group. and fortunately, the community has already stepped up. just to give a few examples, here are a few parts of the ethereum ecosystem that the ethereum foundation and its subsidiaries  have had nothing to do with: augur: a prediction market that has earned $4.5 million in its recent (and still ongoing) crowdsale groupgnosis: another prediction market being developed by consensys which is already processing bets on the ethereum block difficulty, sports games, and soon presidential elections embark: a nodejs-based dapp development, testing and deployment framework truffle: another dapp development, testing and deployment framework ether.camp: a block explorer etherscan.io: another block explorer tradeblock: did i forget to say there’s another ethereum block explorer? etherex: an ethereum-based asset exchange the ether.camp web-based integrated development environment (coming soon) ethereumwallet.com: an online ether wallet the ethereum java implementation (for which original work was done under the foundation, but which is now continuing completely independently) and the ethereum haskell implementation, this time with none of our involvement at all! myetherwallet: another ether wallet metamask: an ethereum browser-in-a-browser andreas oloffson’s development tutorials the first data feed contract ethereum alarm clock, an implementation of one our major planned features for ethereum 1.1, but as a decentralized middleware service right on the 1.0 ethereum blockchain! dapps.ethercasts.com: a webpage listing many of the above, and more (no, i won't mention the ponzies and gambling sites, except insofar as to credit martin holst swende's wonderful work in documenting the perils of building a blockchain-based casino with a bad random number generator, and qian youcai's ongoing work on randao to make this situation better). actually, the ethereum ecosystem is maturing nicely, and looks unrecognizable from what it was barely a year ago. on the inside, we have ethereum foundation subsidiary developers building yet more block explorers and other tools in their spare time, and some developers are already working on implementing ethereum-based lightning networks, identity and reputation systems, and more. in the near future, there will be several more non-profit and for-profit entities emerging in and around the space, some with the involvement of ethereum team members, and many with partial involvement from myself. the first of these to announce itself is the wanxiang blockchain research institute and fund based in shanghai (yes, this is the “major collaboration” i hinted on recently, and is also my much delayed answer to “how did your china trip go?”), which includes (i) an agreement to purchase 416k eth, which has already concluded, (ii) an upcoming conference in october, (iii) a non-profit blockchain research institute, and (iv) a $50m blockchain venture-capital fund, all with emphasis on ethereum development. i fully expect that within six months the ethereum for-profit ecosystem may well be much more well-capitalized than the foundation itself. note that a substantial number of ethereum foundation subsidiary staff is going to be moving over to the rapidly growing for-profit ethereum ecosystem over the next half year in order to bring more funds, interest and development effort into ethereum-land; so far, everyone i have talked to who is leaving the foundation subsidiaries is intending to do this, and they will in many cases simply be continuing, and expanding, the same work that they have started on now either under foundation subsidiary employment or as personal side projects, under a different banner. ming chan, who has recently joined the foundation, will be managing the foundation’s administrative matters, helping to develop an updated and more detailed strategic plan, oversee devcon 1 setup, and generally make sure that things on the foundation side work smoothly throughout the many simultaneous transitions that are taking place; we have also expanded our advisory board, and the new advisors will be announced soon. under these circumstances, we must thus ask, what is the foundation going to do (and not do)? finances let us start off by providing an overview of the foundation's financial situation. its current holdings are roughly: 200,000 chf 1,800 btc 2,700,000 eth plus a 490,000 chf legal fund that will be reserved to cover possible legal defense (it’s like insurance). the foundation's monthly expenditures are currently ~410,000 chf and starting oct 1 are projected to fall to 340,000 chf; a mid-term goal has been placed of 200,000 250,000 chf as a good target that allows us to deliver on our remaining, but important responsibilities. assuming that we get there in three months and that ether and bitcoin prices stay the same (heh), we have enough to last until roughly jun 2016 at the 340,000 rate, and perhaps up to sep-dec 2016 given planned transitions; by that point, the intent is for the foundation to secure alternative revenue sources. possible revenue sources past that point include: developer workshops (including extended in-person “courses”) conference tickets and sponsorships third-party donations and grants (whether to the foundation or to projects that the foundation would otherwise be spending resources on) another action that may be taken is, when ethereum switches to proof of stake, keeping 50% of the old issuance rate active for a year and directing the issuance into some kind of mechanism, perhaps a simple voting scheme or perhaps something more complex incorporating delegated voting, decision markets and potentially other revealed-preference tricks from game theory, in order to pay developers. in any case, our original promise that the issuance rate will not exceed 26.00% per year, and the goal that the eventual final issuance will be much lower (likely 0-3% per year) with proof of stake, will both be kept. we highly welcome community input on whether and how to go down this path; if there is large opposition we will not do this, though the community should understand that not doing this comes with a risk of greater reliance on the for-profit ethereum ecosystem. focus up until perhaps six months ago, the foundation and its subsidiaries have been doing almost everything in the ecosystem; right now, the foundation and its subsidiaries are still doing much of everything though some community members have stepped up to compete with its own offerings in some cases, in my own humble opinion, quite excellently. going forward, the foundation and its subsidiaries will aim for a more focused approach where it carries out only some of the work in the ecosystem, but does it well. an approximate outline of the foundation's activities can be described as follows: education online documentation and developer resources (new documentation site coming soon!) conferences (devcon 1 coming in november!) hackathons, workshops possibly paid in-person development courses conferences, events, meetups co-ordination outreach, marketing and evangelism, both to the media/public and to institutions compliance and regulatory maintenance certifying businesses, individuals, etc (whether ourselves or through a third-party partner) highly targeted core development tasks including: some core client code network observation and coordinating emergency response maintaining test suites, certifying clients paying for some security audits research, including: proof of stake (casper) scalability virtual machine upgrades abstraction formal verification zero-knowledge proof integration official protocol and sub-protocol specifications higher-level development tasks will in the medium term be done largely by for-profit entities, volunteers and other members of the community, although the foundation’s subsidiaries will continue to employ many of the developers in the short term. transparency the ethereum foundation would like to express a renewed interest in being maximally transparent in its affairs; to that end, we are publishing the information above, and as an initial trial in going further we are working with consensys to use their (ethereum) blockchain-based accounting software balanc3 to record all expenses relating to devcon 1. another important aspect of transparency is more open and inclusive development; to that end, we are making a renewed push to move conversations from skype to gitter where they are more publicly visible (eg. you can check out this room right now) and members of the public can more easily participate. we are also evaluating the possibility of introducing a more formal and inclusive process for agreeing on protocol upgrades and welcome input from client developers on this. and there are more announcements both from ourselves and others that will be following soon. in sum, despite the evidence of growing pains, the state of the ethereum nation is good, its ecosystem is vibrant, and its future is bright. as a foundation, we will continue to focus on promoting and supporting research, development and education to bring decentralized protocols and tools to the world that empower developers to produce next generation (d)apps, and together build a more globally accessible, more free and more trustworthy internet. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum protocol update 1 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum protocol update 1 posted by stephan tual on august 4, 2015 protocol announcements here comes the first frontier patch, and this is a big one! before you go further, if your keys date back from go 1.0 rc or c++ 0.9.36, note that you should regenerate all of your keys now. though some releases of the pre-alpha and alpha clients are secure, this depends on which codebase and which version. you should assume that all keys generated prior to these clients are insecure and compromised. this, note, does not affect the pre-sale wallets. thawing the thawing pull request has been merged for the go client and the master branch has been updated. if you're still running branch release 1.0.0 please switch to master. as for the c++ users, they can also switch to master once merged, and binaries will be made available shortly. if you’d rather not update eth but still wish to help thaw the network, you can also just restart eth with an extra parameter of --gas-floor 3141592. i thought that i'd also repost a quick explanation on how the gas limit targeting process operates, and why we cannot guarantee a time by which we'll reach the 21k limit necessary to process one basic transaction per block. each miner runs a copy of geth or eth. in the frontier genesis release, both were set to target 5k and never deviate from that figure. now, with this update, both clients will be updated to instead target 3m gas limit per block. but they cannot switch to a 3m gas limit just like that, because the protocol forces them to climb slowly at a certain rate. that rate is equal to previous block limit / 1024. now, assuming all miners update their clients, and none of them mess around with the settings, we're going to reach 3m within 28h assuming a steady 15s block time including propagation. but here's the thing not all miners are going to update in time, some might forget and some might never update! so, going forward, if a winning block is mined by a updated miner, the block limit will adjust upwards by the rate intended, but if it is mined by a 'lazy' miner who didn't update, it will adjust back downwards (as the lazy miner is still targeting 5k). for this reason, it will take a minimum of 6h to get to a 21k gas limit per block(1 trx per block), and a minimum of 28h to get to 3m. in practice, it will likely take considerably longer than that. this is where the free market come into play. technically, miners could even have colluded a few days ago to modify the client code and make the network behave rather differently than what we had in mind. we merely act as advisers to the community. default, community chosen, genesis block the genesis block we have seen adopted by the community has now been hardcoded in the clients, and you no longer need to specify the --genesis parameter to start eth or geth. that said, you can still specify a hand-crafted genesis block if you want to start a private chain with different genesis, for example. bug fixes on the go client side, a series of bug fixes and improvement have been merged into 1.0.1, including readying ourselves for a go 1.5 release. fixed crash in chain manager #1568 log number fixes #1545 crypto fix for go 1.5 #1536 fix for eth_call and eth_estimategas #1534 console resend method fix #1461 on the c++ client, a full external audit has been carried out on its key store and cryptography. all actions recommended by our expert reviewers have been acted upon. numerous optimizations and security improvements were added to the client: a unique c++ client feature is secure secrets: all heap memory used to store your secrets is securely overwritten immediately after it is finished the c++ client now relies upon boost::random_device, giving a cryptographically secure random-number generation several crash and build fixes for alethzero enhancements to alethzero, especially regarding account information the cli tool ‘ethkey’ can now be used in concert with alethzero to inspect and sign an otherwise unsigned transaction. this allows you to have a completely air-gapped cold-wallet-device difficulty adjustment scheme a lot of you have been wondering how we would implement a switch from pow to pos in time for serenity. this will be handled by the newly introduced difficulty adjustment scheme, which elegantly guarantees a hard-fork point in the next 16 months. it works as follow: starting from block 200,000 (very roughly 17 days from now), the difficulty will undergo an exponential increase which will only become noticeable in about a year. at that point (just around the release of the serenity milestone), we'll see a significant increase in difficulty which will start pushing the block resolution time upwards. so, a year on, the network will continue to be useful for roughly 3-4 months, but eventually will reach an 'ice age' of sorts: the difficulty will simply be too high for anyone to find a block. this will allow us to introduce pos, perhaps via casper, if it proves itself. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle on nathan schneider on the limits of cryptoeconomics 2021 sep 26 see all posts nathan schneider has recently released an article describing his perspectives on cryptoeconomics, and particularly on the limits of cryptoeconomic approaches to governance and what cryptoeconomics could be augmented with to improve its usefulness. this is, of course, a topic that is dear to me ([1] [2] [3] [4] [5]), so it is heartening to see someone else take the blockchain space seriously as an intellectual tradition and engage with the issues from a different and unique perspective. the main question that nathan's piece is trying to explore is simple. there is a large body of intellectual work that criticizes a bubble of concepts that they refer to as "economization", "neoliberalism" and similar terms, arguing that they corrode democratic political values and leave many people's needs unmet as a result. the world of cryptocurrency is very economic (lots of tokens flying around everywhere, with lots of functions being assigned to those tokens), very neo (the space is 12 years old!) and very liberal (freedom and voluntary participation are core to the whole thing). do these critiques also apply to blockchain systems? if so, what conclusions should we draw, and how could blockchain systems be designed to account for these critiques? nathan's answer: more hybrid approaches combining ideas from both economics and politics. but what will it actually take to achieve that, and will it give the results that we want? my answer: yes, but there's a lot of subtleties involved. what are the critiques of neoliberalism and economic logic? near the beginning of nathan's piece, he describes the critiques of overuse of economic logic briefly. that said, he does not go much further into the underlying critiques himself, preferring to point to other sources that have already covered the issue in depth: the economics in cryptoeconomics raises a particular set of anxieties. critics have long warned against the expansion of economic logics, crowding out space for vigorous politics in public life. from the zapatista insurgents of southern mexico (hayden, 2002) to political theorists like william davies (2014) and wendy brown (2015), the "neoliberal" aspiration for economics to guide all aspects of society represents a threat to democratic governance and human personhood itself. here is brown: neoliberalism transmogrifies every human domain and endeavor, along with humans themselves, according to a specific image of the economic. all conduct is economic conduct; all spheres of existence are framed and measured by economic terms and metrics, even when those spheres are not directly monetized. in neoliberal reason and in domains governed by it, we are only and everywhere homo oeconomicus (p. 10) for brown and other critics of neoliberalism, the ascent of the economic means the decline of the political—the space for collective determinations of the common good and the means of getting there. at this point, it's worth pointing out that the "neoliberalism" being criticized here is not the same as the "neoliberalism" that is cheerfully promoted by the lovely folks at the neoliberal project; the thing being critiqued here is a kind of "enough two-party trade can solve everything" mentality, whereas the neoliberal project favors a mix of markets and democracy. but what is the thrust of the critique that nathan is pointing to? what's the problem with everyone acting much more like homo oeconomicus? for this, we can take a detour and peek into the source, wendy brown's undoing the demos, itself. the book helpfully provides a list of the top "four deleterious effects" (the below are reformatted and abridged but direct quotes): intensified inequality, in which the very top strata acquires and retains ever more wealth, the very bottom is literally turned out on the streets or into the growing urban and sub-urban slums of the world, while the middle strata works more hours for less pay, fewer benefits, less security... crass or unethical commercialization of things and activities considered inappropriate for marketization. the claim is that marketization contributes to human exploitation or degradation, [...] limits or stratifies access to what ought to be broadly accessible and shared, [...] or because it enables something intrinsically horrific or severely denigrating to the planet. ever-growing intimacy of corporate and finance capital with the state, and corporate domination of political decisions and economic policy economic havoc wreaked on the economy by the ascendance and liberty of finance capital, especially the destabilizing effects of the inherent bubbles and other dramatic fluctuations of financial markets. the bulk of nathan's article follows along with analyses of how these issues affect daos and governance mechanisms within the crypto space specifically. nathan focuses on three key problems: plutocracy: "those with more tokens than others hold more [i would add, disproportionately more] decision-making power than others..." limited exposure to diverse motivations: "cryptoeconomics sees only a certain slice of the people involved. concepts such as selfsacrifice, duty, and honor are bedrock features of most political and business organizations, but difficult to simulate or approximate with cryptoeconomic incentive design" positive and negative externalities: "environmental costs are classic externalities—invisible to the feedback loops that the system understands and that communicate to its users as incentives ... the challenge of funding"public goods" is another example of an externality and one that threatens the sustainability of crypteconomic systems" the natural questions that arise for me are (i) to what extent do i agree with this critique at all and how it fits in with my own thinking, and (ii) how does this affect blockchains, and what do blockchain protocols need to actually do to avoid these traps? what do i think of the critiques of neoliberalism generally? i disagree with some, agree with others. i have always been suspicious of criticism of "crass and unethical commercialization", because it frequently feels like the author is attempting to launder their own feelings of disgust and aesthetic preferences into grand ethical and political ideologies a sin common among all such ideologies, often the right (random example here) even more than the left. back in the days when i had much less money and would sometimes walk a full hour to the airport to avoid a taxi fare, i remember thinking that i would love to get compensated for donating blood or using my body for clinical trials. and so to me, the idea that such transactions are inhuman exploitation has never been appealing. but at the same time, i am far from a walter block-style defender of all locally-voluntary two-party commerce. i've written up my own viewpoints expressing similar concerns to parts of wendy brown's list in various articles: multiple pieces decrying the evils of vote buying, or even financialized governance generally the importance of public goods funding. failure modes in financial markets due to subtle issues like capital efficiency. so where does my own opposition to mixing finance and governance come from? this is a complicated topic, and my conclusions are in large part a result of my own failure after years of attempts to find a financialized governance mechanism that is economically stable. so here goes... finance is the absence of collusion prevention out of the standard assumptions in what gets pejoratively called "spherical cow economics", people normally tend to focus on the unrealistic nature of perfect information and perfect rationality. but the unrealistic assumption that is hidden in the list that strikes me as even more misleading is individual choice: the idea that each agent is separately making their own decisions, no agent has a positive or negative stake in another agent's outcomes, and there are no "side games"; the only thing that sees each agent's decisions is the black box that we call "the mechanism". this assumption is often used to bootstrap complex contraptions such as the vcg mechanism, whose theoretical optimality is based on beautiful arguments that because the price each player pays only depends on other players' bids, each player has no incentive to make a bid that does not reflect their true value in order to manipulate the price. a beautiful argument in theory, but it breaks down completely once you introduce the possibility that even two of the players are either allies or adversaries outside the mechanism. economics, and economics-inspired philosophy, is great at describing the complexities that arise when the number of players "playing the game" increases from one to two (see the tale of crusoe and friday in murray rothbard's the ethics of liberty for one example). but what this philosophical tradition completely misses is that going up to three players adds an even further layer of complexity. in an interaction between two people, the two can ignore each other, fight or trade. in an interaction between three people, there exists a new strategy: any two of the three can communicate and band together to gang up on the third. three is the smallest denominator where it's possible to talk about a 51%+ attack that has someone outside the clique to be a victim. when there's only two people, more coordination can only be good. but once there's three people, the wrong kind of coordination can be harmful, and techniques to prevent harmful coordination (including decentralization itself) can become very valuable. and it's this management of coordination that is the essence of "politics". going from two people to three introduces the possibility of harms from unbalanced coordination: it's not just "the individual versus the group", it's "the individual versus the group versus the world". now, we can understand try to use this framework to understand the pitfalls of "finance". finance can be viewed as a set of patterns that naturally emerge in many kinds of systems that do not attempt to prevent collusion. any system which claims to be non-finance, but does not actually make an effort to prevent collusion, will eventually acquire the characteristics of finance, if not something worse. to see why this is the case, compare two point systems we are all familiar with: money, and twitter likes. both kinds of points are valuable for extrinsic reasons, both have inevitably become status symbols, and both are number games where people spend a lot of time optimizing to try to get a higher score. and yet, they behave very differently. so what's the fundamental difference between the two? the answer is simple: it's the lack of an efficient market to enable agreements like "i like your tweet if you like mine", or "i like your tweet if you pay me in some other currency". if such a market existed and was easy to use, twitter would collapse completely (something like hyperinflation would happen, with the likely outcome that everyone would run automated bots that like every tweet to claim rewards), and even the likes-for-money markets that exist illicitly today are a big problem for twitter. with money, however, "i send x to you if you send y to me" is not an attack vector, it's just a boring old currency exchange transaction. a twitter clone that does not prevent like-for-like markets would "hyperinflate" into everyone liking everything, and if that twitter clone tried to stop the hyperinflation by limiting the number of likes each user can make, the likes would behave like a currency, and the end result would behave the same as if twitter just added a tipping feature. so what's the problem with finance? well, if finance is optimized and structured collusion, then we can look for places where finance causes problems by using our existing economic tools to understand which mechanisms break if you introduce collusion! unfortunately, governance by voting is a central example of this category; i've covered why in the "moving beyond coin voting governance" post and many other occasions. even worse, cooperative game theory suggests that there might be no possible way to make a fully collusion-resistant governance mechanism. and so we get the fundamental conundrum: the cypherpunk spirit is fundamentally about making maximally immutable systems that work with as little information as possible about who is participating ("on the internet, nobody knows you're a dog"), but making new forms of governance requires the system to have richer information about its participants and ability to dynamically respond to attacks in order to remain stable in the face of actors with unforeseen incentives. failure to do this means that everything looks like finance, which means, well.... perennial over-representation of concentrated interests, and all the problems that come as a result. on the internet, nobody knows if you're 0.0244 of a dog (image source). but what does this mean for governance? the central role of collusion in understanding the difference between kleros and regular courts now, let us get back to nathan's article. the distinction between financial and non-financial mechanisms is key in the article. let us start off with a description of the kleros court: the jurors stood to earn rewards by correctly choosing the answer that they expected other jurors to independently select. this process implements the "schelling point" concept in game theory (aouidef et al., 2021; dylag & smith, 2021). such a jury does not deliberate, does not seek a common good together; its members unite through self-interest. before coming to the jury, the factual basis of the case was supposed to come not from official organs or respected news organizations but from anonymous users similarly disciplined by reward-seeking. the prediction market itself was premised on the supposition that people make better forecasts when they stand to gain or lose the equivalent of money in the process. the politics of the presidential election in question, here, had been thoroughly transmuted into a cluster of economies. the implicit critique is clear: the kleros court is ultimately motivated to make decisions not on the basis of their "true" correctness or incorrectness, but rather on the basis of their financial interests. if kleros is deciding whether biden or trump won the 2020 election, and one kleros juror really likes trump, precommits to voting in his favor, and bribes other jurors to vote the same way, other jurors are likely to fall in line because of kleros's conformity incentives: jurors are rewarded if their vote agrees with the majority vote, and penalized otherwise. the theoretical answer to this is the right to exit: if the majority of kleros jurors vote to proclaim that trump won the election, a minority can spin off a fork of kleros where biden is considered to have won, and their fork may well get a higher market price than the original. sometimes, this actually works! but, as nathan points out, it is not always so simple: but exit may not be as easy as it appears, whether it be from a social-media network or a protocol. the persistent dominance of early-to-market blockchains like bitcoin and ethereum suggests that cryptoeconomics similarly favors incumbency. but alongside the implicit critique is an implicit promise: that regular courts are somehow able to rise above self-interest and "seek a common good together" and thereby avoid some of these failure modes. what is it that financialized kleros courts lack, but non-financialized regular courts retain, that makes them more robust? one possible answer is that courts lack kleros's explicit conformity incentive. but if you just take kleros as-is, remove the conformity incentive (say, there's a reward for voting that does not depend on how you vote), and do nothing else, you risk creating even more problems. kleros judges could get lazy, but more importantly if there's no incentive at all to choose how you vote, even the tiniest bribe could affect a judge's decision. so now we get to the real answer: the key difference between financialized kleros courts and non-financialized regular courts is that financialized kleros courts are, well... financialized. they make no effort to explicitly prevent collusion. non-financialized courts, on the other hand, do prevent collusion in two key ways: bribing a judge to vote in a particular way is explicitly illegal the judge position itself is non-fungible. it gets awarded to specific carefully-selected individuals, and they cannot simply go and sell or reallocate their entire judging rights and salary to someone else. the only reason why political and legal systems work is that a lot of hard thinking and work has gone on behind the scenes to insulate the decision-makers from extrinsic incentives, and punish them explicitly if they are discovered to be accepting incentives from the outside. the lack of extrinsic motivation allows the intrinsic motivation to shine through. furthermore, the lack of transferability allows governance power to be given to specific actors whose intrinsic motivations we trust, avoiding governance power always flowing to "the highest bidder". but in the case of kleros, the lack of hostile extrinsic motivation cannot be guaranteed, and transferability is unavoidable, and so overpoweringly strong in-mechanism extrinsic motivation (the conformity incentive) was the best solution they could find to deal with the problem. and of course, the "final backstop" that kleros relies on, the right of users to fork away, itself depends on social coordination to take place a messy and difficult institution, often derided by cryptoeconomic purists as "proof of social media", that works precisely because public discussion has lots of informal collusion detection and prevention all over the place. collusion in understanding dao governance issues but what happens when there is no single right answer that they can expect voters to converge on? this is where we move away from adjudication and toward governance (yes, i know that adjudication has unavoidably grey edge cases too. governance just has them much more often). nathan writes: governance by economics is nothing new. joint-stock companies conventionally operate on plutocratic governance—more shares equals more votes. this arrangement is economically efficient for aligning shareholder interests (davidson and potts, this issue), even while it may sideline such externalities as fair wages and environmental impacts... in my opinion, this actually concedes too much! governance by economics is not "efficient" once you drop the spherical-cow assumption of no collusion, because it is inherently vulnerable to 51% of the stakeholders colluding to liquidate the company and split its resources among themselves. the only reason why this does not happen much more often "in real life" is because of many decades of shareholder regulation that have been explicitly built up to ban the most common types of abuses. this regulation is, of course, non-"economic" (or, in my lingo, it makes corporate governance less financialized), because it's an explicit attempt to prevent collusion. notably, nathan's favored solutions do not try to regulate coin voting. instead, they try to limit the harms of its weaknesses by combining it with additional mechanisms: rather than relying on direct token voting, as other protocols have done, the graph uses a board-like mediating layer, the graph council, on which the protocol's major stakeholder groups have representatives. in this case, the proposal had the potential to favor one group of stakeholders over others, and passing a decision through the council requires multiple stakeholder groups to agree. at the same time, the snapshot vote put pressure on the council to implement the will of token-holders. in the case of 1hive, the anti-financialization protections are described as being purely cultural: according to a slogan that appears repeatedly in 1hive discussions, "come for the honey, stay for the bees." that is, although economics figures prominently as one first encounters and explores 1hive, participants understand the community's primary value as interpersonal, social, and non-economic. i am personally skeptical of the latter approach: it can work well in low-economic-value communities that are fun oriented, but if such an approach is attempted in a more serious system with widely open participation and enough at stake to invite determined attack, it will not survive for long. as i wrote above, "any system which claims to be non-finance, but does not actually make an effort to prevent collusion, will eventually acquire the characteristics of finance". [edit/correction 2021.09.27: it has been brought to my attention that in addition to culture, financialization is limited by (i) conviction voting, and (ii) juries enforcing a covenant. i'm skeptical of conviction voting in the long run; many daos use it today, but in the long term it can be defeated by wrapper tokens. the covenant, on the other hand, is interesting. my fault for not checking in more detail.] the money is called honey. but is calling money honey enough to make it work differently than money? if not, how much more do you have to do? the solution in thegraph is very much an instance of collusion prevention: the participants have been hand-picked to come from diverse constituencies and to be trusted and upstanding people who are unlikely to sell their voting rights. hence, i am bullish on that approach if it successfully avoids centralization. so how can we solve these problems more generally? nathan's post argues: a napkin sketch of classical, never-quite-achieved liberal democracy (brown, 2015) would depict a market (governed through economic incentives) enclosed in politics (governed through deliberation on the common good). economics has its place, but the system is not economics all the way down; the rules that guide the market, and that enable it in the first place, are decided democratically, on the basis of citizens' civil rights rather than their economic power. by designing democracy into the base-layer of the system, it is possible to overcome the kinds of limitations that cryptoeconomics is vulnerable to, such as by counteracting plutocracy with mass participation and making visible the externalities that markets might otherwise fail to see. there is one key difference between blockchain political theory and traditional nation-state political theory and one where, in the long run, nation states may well have to learn from blockchains. nation-state political theory talks about "markets embedded in democracy" as though democracy is an encompassing base layer that encompasses all of society. in reality, this is not true: there are multiple countries, and every country at least to some degree permits trade with outside countries whose behavior they cannot regulate. individuals and companies have choices about which countries they live in and do business in. hence, markets are not just embedded in democracy, they also surround it, and the real world is a complicated interplay between the two. blockchain systems, instead of trying to fight this interconnectedness, embrace it. a blockchain system has no ability to regular "the market" in the sense of people's general ability to freely make transactions. but what it can do is regulate and structure (or even create) specific markets, setting up patterns of specific behaviors whose incentives are ultimately set and guided by institutions that have anti-collusion guardrails built in, and can resist pressure from economic actors. and indeed, this is the direction nathan ends up going in as well. he talks positively about the design of civil as an example of precisely this spirit: the aborted ethereum-based project civil sought to leverage cryptoeconomics to protect journalism against censorship and degraded professional standards (schneider, 2020). part of the system was the civil council, a board of prominent journalists who served as a kind of supreme court for adjudicating the practices of the network's newsrooms. token holders could earn rewards by successfully challenging a newsroom's practices; the success or failure of a challenge ultimately depended on the judgment of the civil council, designed to be free of economic incentives clouding its deliberations. in this way, a cryptoeconomic enforcement market served a non-economic social mission. this kind of design could enable cryptoeconomic networks to serve purposes not reducible to economic feedback loops. this is fundamentally very similar to an idea that i proposed in 2018: prediction markets to scale up content moderation. instead of doing content moderation by running a low-quality ai algorithm on all content, with lots of false positives, there could be an open mini prediction market on each post, and if the volume got high enough a high-quality committee could step in an adjudicate, and the prediction market participants would be penalized or rewarded based on whether or not they had correctly predicted the outcome. in the mean time, posts with prediction market scores predicting that the post would be removed would not be shown to users who did not explicitly opt-in to participate in the prediction game. there is precedent for this kind of open but accountable moderation: slashdot meta moderation is arguably a limited version of it. this more financialized version of meta-moderation through prediction markets could produce superior outcomes because the incentives invite highly competent and professional participants to take part. nathan then expands: i have argued that pairing cryptoeconomics with political systems can help overcome the limitations that bedevil cryptoeconomic governance alone. introducing purpose-centric mechanisms and temporal modulation can compensate for the blind-spots of token economies. but i am not arguing against cryptoeconomics altogether. nor am i arguing that these sorts of politics must occur in every app and protocol. liberal democratic theory permits diverse forms of association and business within a democratic structure, and similarly politics may be necessary only at key leverage points in an ecosystem to overcome the limitations of cryptoeconomics alone. this seems broadly correct. financialization, as nathan points out in his conclusion, has benefits in that it attracts a large amount of motivation and energy into building and participating in systems that would not otherwise exist. furthermore, preventing financialization is very difficult and high cost, and works best when done sparingly, where it is needed most. however, it is also true that financialized systems are much more stable if their incentives are anchored around a system that is ultimately non-financial. prediction markets avoid the plutocracy issues inherent in coin voting because they introduce individual accountability: users who acted in favor of what ultimately turns out to be a bad decision suffer more than users who acted against it. however, a prediction market requires some statistic that it is measuring, and measurement oracles cannot be made secure through cryptoeconomics alone: at the very least, community forking as a backstop against attacks is required. and if we want to avoid the messiness of frequent forks, some other explicit non-financialized mechanism at the center is a valuable alternative. conclusions in his conclusion, nathan writes: but the autonomy of cryptoeconomic systems from external regulation could make them even more vulnerable to runaway feedback loops, in which narrow incentives overpower the common good. the designers of these systems have shown an admirable capacity to devise cryptoeconomic mechanisms of many kinds. but for cryptoeconomics to achieve the institutional scope its advocates hope for, it needs to make space for less-economic forms of governance. if cryptoeconomics needs a political layer, and is no longer self-sufficient, what good is cryptoeconomics? one answer might be that cryptoeconomics can be the basis for securing more democratic and values-centered governance, where incentives can reduce reliance on military or police power. through mature designs that integrate with less-economic purposes, cryptoeconomics might transcend its initial limitations. politics needs cryptoeconomics, too ... by integrating cryptoeconomics with democracy, both legacies seem poised to benefit. i broadly agree with both conclusions. the language of collusion prevention can be helpful for understanding why cryptoeconomic purism so severely constricts the design space. "finance" is a category of patterns that emerge when systems do not attempt to prevent collusion. when a system does not prevent collusion, it cannot treat different individuals differently, or even different numbers of individuals differently: whenever a "position" to exert influence exists, the owner of that position can just sell it to the highest bidder. gavels on amazon. a world where these were nfts that actually came with associated judging power may well be a fun one, but i would certainly not want to be a defendant! the language of defense-focused design, on the other hand, is an underrated way to think about where some of the advantages of blockchain-based designs can be. nation state systems often deal with threats with one of two totalizing mentalities: closed borders vs conquer the world. a closed borders approach attempts to make hard distinctions between an "inside" that the system can regulate and an "outside" that the system cannot, severely restricting flow between the inside and the outside. conquer-the-world approaches attempt to extraterritorialize a nation state's preferences, seeking a state of affairs where there is no place in the entire world where some undesired activity can happen. blockchains are structurally unable to take either approach, and so they must seek alternatives. fortunately, blockchains do have one very powerful tool in their grasp that makes security under such porous conditions actually feasible: cryptography. cryptography allows everyone to verify that some governance procedure was executed exactly according to the rules. it leaves a verifiable evidence trail of all actions, though zero knowledge proofs allow mechanism designers freedom in picking and choosing exactly what evidence is visible and what evidence is not. cryptography can even prevent collusion! blockchains allow applications to live on a substrate that their governacne does not control, which allows them to effectively implement techniques such as, for example, ensure that every change to the rules only takes effect with a 60 day delay. finally, freedom to fork is much more practical, and forking is much lower in economic and human cost, than most centralized systems. blockchain-based contraptions have a lot to offer the world that other kinds of systems do not. on the other hand, nathan is completely correct to emphasize that blockchainized should not be equated with financialized. there is plenty of room for blockchain-based systems that do not look like money, and indeed we need more of them. security advisory [eth (cpp-ethereum) potentially vulnerable if running with upnp enabled] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security advisory [eth (cpp-ethereum) potentially vulnerable if running with upnp enabled] posted by gustav simonsson on october 10, 2015 security affected configurations: issue reported for eth (cpp-ethereum).likelihood: mediumseverity: highimpact: potentially achieve remote code execution on a machine running eth (cpp-ethereum)details:a vulnerability found in the miniupnp library can potentially affect eth clients running with upnp enabled. effects on expected chain reorganisation depth: noneremedial action taken by ethereum: we are verifying whether this can indeed affect cpp-ethereum and will post an update shortly.proposed temporary workaround: only run eth (cpp-ethereum) with upnp disabledby passing --upnp off to eth.advisory: disable upnp if running the eth client (cpp-ethereum). previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements on anti-pre-revelation games | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on anti-pre-revelation games posted by vitalik buterin on august 28, 2015 research & development an increasing number of proposed applications on top of ethereum rely on some kind of incentivized, multi-party data provision whether voting, random number collection, or other use cases where getting information from multiple parties to increase decentralization is highly desirable, but also where there is a strong risk of collusion. a randao can certainly provide random numbers with much higher cryptoeconomic security than simple block hashes and certainly better than deterministic algorithms with publicly knowable seeds, but it is not infinitely collusion-proof: if 100% of participants in a randao collude with each other, they can set the result to whatever they want. a much more controversial example is the prediction market augur, where decentralized event reporting relies on a highly advanced version of a schelling scheme, where everyone votes on the result and everyone in the majority gets rewarded. the theory is that if you expect everyone else to be honest, your incentive is also to be honest to be in the majority, and so honesty is a stable equilibrium; the problem is, however, that is more than 50% of the participants collude, the system breaks. the fact that augur has an independent token provides a partial defense against this problem: if the voters collude, then the value of augur's token can be expected to decrease to near-zero as the system becomes perceived as useless and unreliable, and so the colluders lose a large amount of value. however, it is certainly not a total defense. paul sztorc's truthcoin (and also augur) includes a further defense, which is quite economically clever. the core mechanism is simple: rather than simply awarding a static amount to everyone in the majority, the amount awarded depends on the level of disagreement among the final votes, and the more disagreement there is the more majority voters get, and minority voters get an equally large amount taken out of their security deposit. the intent is simple: if you get a message from someone saying "hey, i am starting a collusion; even though the actual answer is a, let's all vote b", in a simpler scheme you may be inclined to go along. in sztorc's scheme, however, you may well come to the conclusion that this individual is actually going to vote a, and is trying to convince only a few percent of people to vote b, so as to steal some of their money. hence, it creates a lack of trust, making collusions harder. however, there is a problem: precisely because blockchains are such excellent devices for cryptographically secure agreements and coordination, it's very hard to make it impossible to collude provably. to see how, consider the simplest possible scheme for how reporting votes in augur might work: there is a period during which everyone can send a transaction supplying their vote, and at the end the algorithm calculates the result. however, this approach is fatally flawed: it creates an incentive for people to wait as long as possible to see what all the other players' answers are before answering themselves. taking this to its natural equilibrium, we would have everyone voting in the last possible block, leading to the miner of the last block essentially controlling everything. a scheme where the end comes randomly (eg. the first block that passes 100x the usual difficulty threshold) mitigates this somewhat, but still leaves a great amount of power in the hands of individual miners. the standard cryptographer's response to this problem is the hash-commit-reveal scheme: every player p[i] determines their response r[i], and there is a period during which everyone must submit h(r[i]) where h can be any pre-specified hash function (eg. sha3). after that, everyone must submit r[i], and the values are checked against the previously provided hashes. for two-player rock paper scissors, or any other game which is purely zero-sum, this works great. for augur, however, it still leaves open the opportunity for credible collusion: users can voluntarily reveal r[i] before the fact, and others can check that this indeed matches the hash values that they provided to the chain. allowing users to change their hashes before the hash submitting period runs out does nothing; users can always lock up a large amount of money in a specially crafted contract that only releases it if no one provides a merkle tree proof to the contract, culminating with a previous blockhash, showing that the vote was changed, thereby committing not to change their vote. a new solution? however, there is also another path to solving this problem, one that has not yet been adequately explored. the idea is this: instead of making pre-revelation for collusion purposes costly within the primary game itself, we introduce a parallel game (albeit a mandatory one, backed by the oracle participants' security deposits) where anyone who pre-reveals any information about their vote to anyone else opens themselves up to the risk of being (probabilistically) betrayed, without any way to prove that it was that specific person who betrayed them. the game, in its most basic form, works as follows. suppose that there is a decentralized random number generation scheme where users must all flip a coin and supply either 0 or 1 as inputs. now, suppose that we want to disincentivize collusion. what we do is simple: we allow anyone to register a bet against any player in the system (note the use of "anyone" and "any player"; non-players can join as long as they supply the security deposit), essentially stating "i am confident that this person will vote x with more than 1/2 probability", where x can be 0 or 1. the rules of the bet are simply that if the target supplies x as their input then n coins are transferred from them to the bettor, and if the target supplies the other value then n coins are transferred from the bettor to the target. bets can be made in an intermediate phase between commitment and revelation. probabilistically speaking, any provision of information to any other party is now potentially extremely costly; even if you convince someone else that you will vote 1 with 51% probability, they can still take coins from you probabilistically, and they will win out in the long run as such a scheme gets repeated. note that the other party can bet anonymously, and so can always pretend that it was a passerby gambler making the bets, and not them. to enhance the scheme further, we can say that you must bet against n different players at the same time, and the players must be pseudorandomly selected from a seed; if you want to target a specific player, you can do so by trying different seeds until you get your desired target alongside a few others, but there will always be at least some plausible deniability. another possible enhancement, though one which has its costs, is to require players to only register their bets between commitment and revelation, only revealing and executing the bets long after many rounds of the game have taken place (we assume that there is a long period before security deposits can be taken out for this to work). now, how do we convert this into the oracle scenario? consider once again the simple binary case: users report either a or b, and some portion p, unknown before the end of the process, will report a and the remaining 1-p will report b. here, we change the scheme somewhat: the bets now say "i am confident that this person will vote x with more than p probability". note that the language of the bet should not be taken to imply knowledge of p; rather, it implies an opinion that, whatever the probability a random user will vote x is, the one particular user that the bettor is targeting will vote x with higher probability than that. the rules of the bet, processed after the voting phase, are that if the target votes x then n * (1 p) coins are transferred from the target to the bettor, and otherwise n * p coins are transferred from the bettor to the target. note that, in the normal case, profit here is even more guaranteed than it is in the binary randao example above: most of the time, if a is the truth, everyone votes for a, so the bets would be very low-risk profit grabs even if complex zero-knowledge-proof protocols were used to only give probabilistic assurance that they will vote for a particular value. side technical note: if there are only two possibilities, then why can't you determine r[i] from h(r[i]) just by trying both options? the answer is that users are actually publishing h(r[i], n) and (r[i], n) for some large random nonce n that will get discarded, so there is too much space to enumerate. as another point, note that this scheme is in a sense a superset of paul sztorc's counter-coordination scheme described above: if someone convinces someone else to falsely vote b when the real answer is a, then they can bet against them with this information secretly. particularly, profiting from others' moral turpitude would now be no longer a public good, but rather a private good: an attacker that tricks someone else into a false collusion could gain 100% of the profit, so there would be even more suspicion to join a collusion that's not cryptographically provable. now, how does this work in the linear case? suppose that users are voting on the btc/usd price, so they need to supply not a choice between a and b, but rather a scalar value. the lazy solution is simply to apply the binary approach in parallel to every binary digit of the price; an alternative solution, however, is range betting. users can make bets of the form "i am confident that this person will vote between x and y with higher probability than the average person"; in this way, revealing even roughly what value you are going to be voting to anyone else is likely to be costly. problems what are the weaknesses of the scheme? perhaps the largest one is that it opens up an opportunity to "second-order grief" other players: although one cannot, in expectation, force other players to lose money to this scheme, one can certainly expose them to risk by betting against them. hence, it may open up opportunities for blackmail: "do what i want or i'll force you to gamble with me". that said, this attack does come at the cost of the attacker themselves being subjected to risk. the simplest way to mitigate this is to limit the amount that can be gambled, and perhaps even limit it in proportion to how much is bet. that is, if p = 0.1, allow bets up to $1 saying "i am confident that this person will vote x with more than 0.11 probability", bets up to $2 saying "i am confident that this person will vote x with more than 0.12 probability", etc (mathematically advanced users may note that devices like logarithmic market scoring rules are good ways of efficiently implementing this functionality); in this case, the amount of money you can extract from someone will be quadratically proportional to the level of private information that you have, and performing large amounts of griefing is in the long run guaranteed to cost the attacker money, and not just risk. the second is that if users are known to be using multiple particular sources of information, particularly on more subjective questions like "vote on the price of token a / token b" and not just binary events, then those users will be exploitable; for example, if you know that some users have a history of listening to bitstamp and some to bitfinex to get their vote information, then as soon as you get the latest feeds from both exchanges you can probabilistically extract some amount of money from a participant based on your estimation of which exchange they are listening to. hence, it remains a research problem to see exactly how users would respond in that case. note that such events are a complicated issue in any case; failure modes such as everyone centralizing on one particular exchange are very likely to arise even in simple sztorcian schemes without this kind of probabilistic griefing. perhaps a multi-layered scheme with a second-layer "appeals court" of voting at the top that is invoked so rarely that the centralization effects never end up taking place may mitigate the problem, but it remains a highly empirical question. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements announcing eπ: ethereum on raspberry pi programme | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search announcing eπ: ethereum on raspberry pi programme posted by gavin wood on may 14, 2015 research & development as part of our drive to introduce ethereum to the far corners of the world, not to mention demonstrate the potential for embedded blockchain technology (even without light-nodes) i'm happy to announce the roll-out of our "eπ" ethereum-on-raspberry pi programme. as part of our programme, we're offering a full-kit for placing a full ethereum node on the internet to communities around the world. included in the kit is a raspberry pi (version 2), power connector and sd card preloaded with all software required to run an ethereum frontier node. the only thing that must be supplied by the applicant is power and internet. before you ask, no they won't make a good miner (aside from anything else, they don't have nearly enough gpu ram). however, they'll act as archive nodes, verify and transmit blocks and transactions and generally add to the party atmosphere. we have quite a few of these to give away so please use our online form. everyone is welcome to apply but we're especially looking for schools, universities and communities in the oft-forgotten far reaches of the world. if you don't make it in to the programme, but fancy setting up a low-cost node anyway (these devices are around $50 each) then christoph has helpfully provided a howto, to get you on your way. in other news, olympic is whirring away and there still plenty of ether to be won by people willing to mine or code a few scripts to hammer the network with transactions. i've set up an ether faucet on olympic (named as such in the ethereum name registry), so go ahead and bag yourself a bit of free ether! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle don't overload ethereum's consensus 2023 may 21 see all posts special thanks to karl floersch and justin drake for feedback and review the ethereum network's consensus is one of the most highly secured cryptoeconomic systems out there. 18 million eth (~$34 billion) worth of validators finalize a block every 6.4 minutes, running many different implementations of the protocol for redundancy. and if the cryptoeconomic consensus fails, whether due to a bug or an intentional 51% attack, a vast community of many thousands of developers and many more users are watching carefully to make sure the chain recovers correctly. once the chain recovers, protocol rules ensure that attackers will likely be heavily penalized. over the years there have been a number of ideas, usually at the thought experiment stage, to also use the ethereum validator set, and perhaps even the ethereum social consensus, for other purposes: the ultimate oracle: a proposal where users can vote on what facts are true by sending eth, with a schellingcoin mechanism: everyone who sent eth to vote for the majority answer gets a proportional share of all the eth sent to vote for the minority answer. the description continues: "so in principle this is an symmetric game. what breaks the symmetry is that a) the truth is the natural point to coordinate on and more importantly b) the people betting on the truth can make a credible thread of forking ethereum if they loose." re-staking: a set of techniques, used by many protocols including eigenlayer, where ethereum stakers can simultaneously use their stake as a deposit in another protocol. in some cases, if they misbehave according to the other protocol's rules, their deposit also gets slashed. in other cases, there are no in-protocol incentives and stake is simply used to vote. l1-driven recovery of l2 projects: it has been proposed on many occasions that if an l2 has a bug, the l1 could fork to recover it. one recent example is this design for using l1 soft forks to recover l2 failures. the purpose of this post will be to explain in detail the argument why, in my view, a certain subset of these techniques brings high systemic risks to the ecosystem and should be discouraged and resisted. these proposals are generally made in a well-intentioned way, and so the goal is not to focus on individuals or projects; rather, the goal is to focus on techniques. the general rule of thumb that this post will attempt to defend is as follows: dual-use of validator staked eth, while it has some risks, is fundamentally fine, but attempting to "recruit" ethereum social consensus for your application's own purposes is not. examples of the distinction between re-using validators (low-risk) and overloading social consensus (high-risk) alice creates a web3 social network where if you cryptographically prove that you control the key of an active ethereum validator, you automatically gain "verified" status. low-risk. bob cryptographically proves that he controls the key of ten active ethereum validators as a way of proving that he has enough wealth to satisfy some legal requirement. low-risk. charlie claims to have disproven the twin primes conjecture, and claims to know the largest p such that p and p+2 are both prime. he changes his staking withdrawal address to a smart contract where anyone can submit a claimed counterexample q > p, along with a snark proving that q and q+2 are both prime. if someone makes a valid claim, then charlie's validator is forcibly exited, and the submitter gets whatever of charlie's eth is left. low-risk. dogecoin decides to switch to proof of stake, and to increase the size of its security pool it allows ethereum stakers to "dual-stake" and simultaneously join its validator set. to do so, ethereum stakers would have to change their staking withdrawal address to a smart contract where anyone can submit a proof that they violated the dogecoin staking rules. if someone does submit such a proof, then the staker's validator is forcibly exited, and whatever of their eth is left is used to buy-and-burn doge. low-risk. ecash does the same as dogecoin, but the project leaders further announce: if the majority of participating eth validators collude to censor ecash transactions, they expect that the ethereum community will hard-fork to delete those validators. they argue that it will be in ethereum's interest to do so as those validators are proven to be malicious and unreliable. high-risk. fred creates an eth/usd price oracle, which functions by allowing ethereum validators to participate and vote. there are no incentives. low-risk. george creates an eth/usd price oracle, which functions by allowing eth holders to participate and vote. to protect against laziness and creeping bribes, they add an incentive mechanism where the participants that give an answer within 1% of the median answer get 1% of the eth of any participants that gave an answer further than 1% from the median. when asked "what if someone credibly offers to bribe all the participants, everyone starts submitting the wrong answer, and so honest people get 10 million of their eth taken away?", george replies: then ethereum will have to fork out the bad participants' money. high-risk. george conspicuously stays away from making replies. medium-high risk (as the project could create incentives to attempt such a fork, and hence the expectation that it will be attmpted, even without formal encouragement) george replies: "then the attacker wins, and we'll give up on using this oracle". medium-low risk (not quite "low" only because the mechanism does create a large set of actors who in a 51% attack might be incentivized to indepently advocate for a fork to protect their deposits) hermione creates a successful layer 2, and argues that because her layer 2 is the largest, it is inherently the most secure, because if there is a bug that causes funds to be stolen, the losses will be so large that the community will have no choice but to fork to recover the users' funds. high-risk. if you're designing a protocol where, even if everything completely breaks, the losses are kept contained to the validators and users who opted in to participating in and using your protocol, this is low-risk. if, on the other hand, you have the intent to rope in the broader ethereum ecosystem social consensus to fork or reorg to solve your problems, this is high-risk, and i argue that we should strongly resist all attempts to create such expectations. a middle ground is situations that start off in the low-risk category but give their participants incentives to slide into the higher-risk category; schellingcoin-style techniques, especially mechanisms with heavy penalties for deviating from the majority, are a major example. so what's so wrong with stretching ethereum consensus, anyway? it is the year 2025. frustrated with the existing options, a group has decided to make a new eth/usd price oracle, which works by allowing validators to vote on the price every hour. if a validator votes, they would be unconditionally rewarded with a portion of fees from the system. but soon participants became lazy: they connected to centralized apis, and when those apis got cyber-attacked, they either dropped out or started reporting false values. to solve this, incentives were introduced: the oracle also votes retrospectively on the price one week ago, and if your (real time or retrospective) vote is more than 1% away from the median retrospective vote, you are heavily penalized, with the penalty going to those who voted "correctly". within a year, over 90% of validators are participating. someone asked: what if lido bands together with a few other large stakers to 51% attack the vote, forcing through a fake eth/usd price value, extracting heavy penalties from everyone who does not participate in the attack? the oracle's proponents, at this point heavily invested in the scheme, reply: well if that happens, ethereum will surely fork to kick the bad guys out. at first, the scheme is limited to eth/usd, and it appears resilient and stable. but over the years, other indices get added: eth/eur, eth/cny, and eventually rates for all countries in the g20. but in 2034, things start to go wrong. brazil has an unexpectedly severe political crisis, leading to a disputed election. one political party ends up in control of the capital and 75% of the country, but another party ends up in control of some northern areas. major western media argue that the northern party is clearly the legitimate winner because it acted legally and the southern party acted illegally (and by the way are fascist). indian and chinese official sources, and elon musk, argue that the southern party has actual control of most of the country, and the international community should not try to be a world police and should instead accept the outcome. by this point, brazil has a cbdc, which splits into two forks: the (northern) brl-n, and the (southern) brl-s. when voting in the oracle, 60% of ethereum stakers provide the eth/brl-s rate. major community leaders and businesses decry the stakers' craven capitulation to fascism, and propose to fork the chain to only include the "good stakers" providing the eth/brl-n rate, and drain the other stakers' balances to near-zero. within their social media bubble, they believe that they will clearly win. however, once the fork hits, the brl-s side proves unexpectedly strong. what they expected to be a landslide instead proves to be pretty much a 50-50 community split. at this point, the two sides are in their two separate universes with their two chains, with no practical way of coming back together. ethereum, a global permissionless platform created in part to be a refuge from nations and geopolitics, instead ends up cleaved in half by any one of the twenty g20 member states having an unexpectedly severe internal issue. that's a nice scifi story you got there. could even make a good movie. but what can we actually learn from it? a blockchain's "purity", in the sense of it being a purely mathematical construct that attempts to come to consensus only on purely mathematical things, is a huge advantage. as soon as a blockchain tries to "hook in" to the outside world, the outside world's conflicts start to impact on the blockchain too. given a sufficiently extreme political event in fact, not that extreme a political event, given that the above story was basically a pastiche of events that have actually happened in various major (>25m population) countries all within the past decade even something as benign as a currency oracle could tear the community apart. here are a few more possible scenarios: one of the currencies that the oracle tracks (which could even be usd) simply hyperinflates, and markets break down to the point that at some points in time there is no clear specific market price. if ethereum adds a price oracle to another cryptocurrency, then a controversial split like in the story above is not hypothetical: it's something that has already happened, including in the histories of both bitcoin and ethereum itself. if strict capital controls become operational, then which price to report as the legitimate market price between two currencies becomes a political question. but more importantly, i'd argue that there is a schelling fence at play: once a blockchain starts incorporating real-world price indices as a layer-1 protocol feature, it could easily succumb to interpreting more and more real-world information. introducing layer-1 price indices also expands the blockchain's legal attack surface: instead of being just a neutral technical platform, it becomes much more explicitly a financial tool. what about risks from examples other than price indices? any expansion of the "duties" of ethereum's consensus increases the costs, complexities and risks of running a validator. validators become required to take on the human effort of paying attention and running and updating additional software to make sure that they are acting correctly according to whatever other protocols are being introduced. other communities gain the ability to externalize their dispute resolution needs onto the ethereum community. validators and the ethereum community as a whole become forced to make far more decisions, each of which has some risk of causing a community split. even if there is no split, the desire to avoid such pressure creates additional incentives to externalize the decisions to centralized entities through stake-pooling. the possibility of a split would also greatly strengthen perverse too-big-to-fail mechanics. there are so many layer-2 and application-layer projects on ethereum that it would be impractical for ethereum social consensus to be willing to fork to solve all of their problems. hence, larger projects would inevitably get a larger chance of getting a bailout than smaller ones. this would in turn lead to larger projects getting a moat: would you rather have your coins on arbitrum or optimism, where if something goes wrong ethereum will fork to save the day, or on taiko, where because it's smaller (and non-western, hence less socially connected to core dev circles), an l1-backed rescue is much less likely? but bugs are a risk, and we need better oracles. so what should we do? the best solutions to these problems are, in my view, case-by-case, because the various problems are inherently so different from each other. some solutions include: price oracles: either not-quite-cryptoeconomic decentralized oracles, or validator-voting-based oracles that explicitly commit to their emergency recovery strategies being something other than appealing to l1 consensus for recovery (or some combination of both). for example, a price oracle could count on a trust assumption that voting participants get corrupted slowly, and so users would have early warning of an attack and could exit any systems that depend on the oracle. such an oracle could intentionally give its reward only after a long delay, so that if that instance of the protocol falls into disuse (eg. because the oracle fails and the community forks toward another version), the participants do not get the reward. more complex truth oracles reporting on facts more subjective than price: some kind of decentralized court system built on a not-quite-cryptoeconomic dao. layer 2 protocols: in the short term, rely on partial training wheels (what this post calls stage 1) in the medium term, rely on multiple proving systems. trusted hardware (eg. sgx) could be included here; i strongly anti-endorse sgx-like systems as a sole guarantor of security, but as a member of a 2-of-3 system they could be valuable. in the longer term, hopefully complex functionalities such as "evm validation" will themselves eventually be enshrined in the protocol cross-chain bridges: similar logic as oracles, but also, try to minimize how much you rely on bridges at all: hold assets on the chain where they originate and use atomic swap protocols to move value between different chains. using the ethereum validator set to secure other chains: one reason why the (safer) dogecoin approach in the list of examples above might be insufficient is that while it does protect against 51% finality-reversion attacks, it does not protect against 51% censorship attacks. however, if you are already relying on ethereum validators, then one possible direction to take is to move away from trying to manage an independent chain entirely, and become a validium with proofs anchored into ethereum. if a chain does this, its protection against finality-reversion attacks becomes as strong as ethereum's, and it becomes secure against censorship up to 99% attacks (as opposed to 49%). conclusions blockchain communities' social consensus is a fragile thing. it's necessary because upgrades happen, bugs happen, and 51% attacks are always a possibility but because it has such a high risk of causing chain splits, in mature communities it should be used sparingly. there is a natural urge to try to extend the blockchain's core with more and more functionality, because the blockchain's core has the largest economic weight and the largest community watching it, but each such extention makes the core itself more fragile. we should be wary of application-layer projects taking actions that risk increasing the "scope" of blockchain consensus to anything other than verifying the core ethereum protocol rules. it is natural for application-layer projects to attempt such a strategy, and indeed such ideas are often simply conceived without appreciation of the risks, but its result can easily become very misaligned with the goals of the community as a whole. such a process has no limiting principle, and could easily lead to a blockchain community having more and more "mandates" over time, pushing it into an uncomfortable choice between a high yearly risk of splitting and some kind of de-facto formalized bureaucracy that has ultimate control of the chain. we should instead preserve the chain's minimalism, support uses of re-staking that do not look like slippery slopes to extending the role of ethereum consensus, and help developers find alternate strategies to achieve their security goals. dark mode toggle halo and more: exploring incremental verification and snarks without pairings 2021 nov 05 see all posts special thanks to justin drake and sean bowe for wonderfully pedantic and thoughtful feedback and review, and to pratyush mishra for discussion that contributed to the original ipa exposition. readers who have been following the zk-snark space closely should by now be familiar with the high level of how zk-snarks work. zk-snarks are based on checking equations where the elements going into the equations are mathematical abstractions like polynomials (or in rank-1 constraint systems matrices and vectors) that can hold a lot of data. there are three major families of cryptographic technologies that allow us to represent these abstractions succinctly: merkle trees (for fri), regular elliptic curves (for inner product arguments (ipas)), and elliptic curves with pairings and trusted setups (for kzg commitments). these three technologies lead to the three types of proofs: fri leads to starks, kzg commitments lead to "regular" snarks, and ipa-based schemes lead to bulletproofs. these three technologies have very distinct tradeoffs: technology cryptographic assumptions proof size verification time fri hashes only (quantum safe!) large (10-200 kb) medium (poly-logarithmic) inner product arguments (ipas) basic elliptic curves medium (1-3 kb) very high (linear) kzg commitments elliptic curves + pairings + trusted setup short (~500 bytes) low (constant) so far, the first and the third have seen the most attention. the reason for this has to do with that pesky right column in the second row of the table: elliptic curve-based inner product arguments have linear verification time. what this means that even though the size of a proof is small, the amount of time needed to verify the proof always takes longer than just running the computation yourself. this makes ipas non-viable for scalability-related zk-snark use cases: there's no point in using an ipa-based argument to prove the validity of an ethereum block, because verifying the proof will take longer than just checking the block yourself. kzg and fri-based proofs, on the other hand, really are much faster to verify than doing the computation yourself, so one of those two seems like the obvious choice. more recently, however, there has been a slew of research into techniques for merging multiple ipa proofs into one. much of the initial work on this was done as part of designing the halo protocol which is going into zcash. these merging techniques are cheap, and a merged proof takes no longer to verify than a single one of the proofs that it's merging. this opens a way forward for ipas to be useful: instead of verifying a size-\(n\) computation with a proof that takes still takes \(o(n)\) time to verify, break that computation up into smaller size-\(k\) steps, make \(\frac{n}{k}\) proofs for each step, and merge them together so the verifier's work goes down to a little more than \(o(k)\). these techniques also allow us to do incremental verification: if new things keep being introduced that need to be proven, you can just keep taking the existing proof, mixing it in with a proof of the new statement, and getting a proof of the new combined statement out. this is really useful for verifying the integrity of, say, an entire blockchain. so how do these techniques work, and what can they do? that's exactly what this post is about. background: how do inner product arguments work? inner product arguments are a proof scheme that can work over many mathematical structures, but usually we focus on ipas over elliptic curve points. ipas can be made over simple elliptic curves, theoretically even bitcoin and ethereum's secp256k1 (though some special properties are preferred to make ffts more efficient); no need for insanely complicated pairing schemes that despite having written an explainer article and an implementation i can still barely understand myself. we'll start off with the commitment scheme, typically called pedersen vector commitments. to be able to commit to degree \(< n\) polynomials, we first publicly choose a set of base points, \(g_0 ... g_{n-1}\). these points can be generated through a pseudo-random procedure that can be re-executed by anyone (eg. the x coordinate of \(g_i\) can be \(hash(i, j)\) for the lowest integer \(j \ge 0\) that produces a valid point); this is not a trusted setup as it does not rely on any specific party to introduce secret information. to commit to a polynomial \(p(x) = \sum_i c_i x^i\), the prover computes \(com(p) = \sum_i c_i g_i\). for example, \(com(x^2 + 4)\) would equal \(g_2 + 4 * g_0\) (remember, the \(+\) and \(*\) here are elliptic curve addition and multiplication). cryptographers will also often add an extra \(r \cdot h\) hiding parameter for privacy, but for simplicity of exposition we'll ignore privacy for now; in general, it's not that hard to add privacy into all of these schemes. though it's not really mathematically accurate to think of elliptic curve points as being like real numbers that have sizes, area is nevertheless a good intuition for thinking about linear combinations of elliptic curve points like we use in these commitments. the blue area here is the value of the pedersen commitment \(c = \sum_i c_i g_i\) to the polynomial \(p = \sum_i c_i x^i\). now, let's get into how the proof works. our final goal will be a polynomial evaluation proof: given some \(z\), we want to make a proof that \(p(z) = a\), where this proof can be verified by anyone who has the commitment \(c = com(p)\). but first, we'll focus on a simpler task: proving that \(c\) is a valid commitment to any polynomial at all that is, proving that \(c\) was constructed by taking a linear combination \(\sum_i c_i g_i\) of the points \(\{g_0 ... g_{n-1}\}\), without anything else mixed in. of course, technically any point is some multiple of \(g_0\) and so it's theoretically a valid commitment of something, but what we care about is proving that the prover knows some \(\{c_0 ... c_{n-1}\}\) such that \(\sum_i c_i g_i = c\). a commitment \(c\) cannot commit to multiple distinct polynomials that the prover knows about, because if it could, that would imply that elliptic curves are broken. the prover could, of course, just provide \(\{c_0 ... c_{n-1}\}\) directly and let the verifier check the commitment. but this takes too much space. so instead, we try to reduce the problem to a smaller problem of half the size. the prover provides two points, \(l\) and \(r\), representing the yellow and green areas in this diagram: you may be able to see where this is going: if you add \(c + l + r\) together (remember: \(c\) was the original commitment, so the blue area), the new combined point can be expressed as a sum of four squares instead of eight. and so now, the prover could finish by providing only four sums, the widths of each of the new squares. repeat this protocol two more times, and we're down to a single full square, which the prover can prove by sending a single value representing its width. but there's a problem: if \(c\) is incorrect in some way (eg. the prover added some extra point \(h\) into it), then the prover could just subtract \(h\) from \(l\) or \(r\) to compensate for it. we plug this hole by randomly scaling our points after the prover provides \(l\) and \(r\): choose a random factor \(\alpha\) (typically, we set \(\alpha\) to be the hash of all data added to the proof so far, including the \(l\) and \(r\), to ensure the verifier can also compute \(\alpha\)). every even \(g_i\) point gets scaled by \(\alpha\), every odd \(g_i\) point gets scaled down by the same factor. every odd coefficient gets scaled up by \(\alpha\) (notice the flip), and every even coefficient gets scaled down by \(\alpha\). now, notice that: the yellow area (\(l\)) gets multiplied by \(\alpha^2\) (because every yellow square is scaled up by \(\alpha\) on both dimensions) the green area (\(r\)) gets divided by \(\alpha^2\) (because every green square is scaled down by \(\alpha\) on both dimensions) the blue area (\(c\)) remains unchanged (because its width is scaled up but its height is scaled down) hence, we can generate our new half-size instance of the problem with some simple transformations: \(g'_{i} = \alpha g_{2i} + \frac{g_{2i+1}}{\alpha}\) \(c'_{i} = \frac{c_{2i}}{\alpha} + \alpha c_{2i+1}\) \(c' = c + \alpha^2 l + \frac{r}{\alpha^2}\) (note: in some implementations you instead do \(g'_i = \alpha g_{2i} + g_{2i+1}\) and \(c'_i = c_{2i} + \alpha c_{2i+1}\) without dividing the odd points by \(\alpha\). this makes the equation \(c' = \alpha c + \alpha^2 l + r\), which is less symmetric, but ensures that the function to compute any \(g'\) in any round of the protocol becomes a polynomial without any division. yet another alternative is to do \(g'_i = \alpha g_{2i} + g_{2i+1}\) and \(c'_i = c_{2i} + \frac{c_{2i+1}}{\alpha}\), which avoids any \(\alpha^2\) terms.) and then we repeat the process until we get down to one point: finally, we have a size-1 problem: prove that the final modified \(c^*\) (in this diagram it's \(c'''\) because we had to do three iterations, but it's \(log(n)\) iterations generally) equals the final modified \(g^*_0\) and \(c^*_0\). here, the prover just provides \(c^*_0\) in the clear, and the verifier checks \(c^*_0 g^*_0 = c^*\). computing \(c^*_0\) required being able to compute a linear combination of \(\{c_0 ... c_{n-1}\}\) that was not known ahead of time, so providing it and verifying it convinces the verifier that the prover actually does know all the coefficients that go into the commitment. this concludes the proof. recapping: the statement we are proving is that \(c\) is a commitment to some polynomial \(p(x) = \sum_i c_i x^i\) committed to using the agreed-upon base points \(\{g_0 ... g_{n-1}\}\) the proof consists of \(log(n)\) pairs of \((l, r)\) values, representing the yellow and green areas at each step. the prover also provides the final \(c^*_0\) the verifier walks through the proof, generating the \(\alpha\) value at each step using the same algorithm as the prover and computing the new \(c'\) and \(g'_i\) values (the verifier doesn't know the \(c_i\) values so they can't compute any \(c'_i\) values) at the end, they check whether or not \(c^*_0 g^*_0 = c^*\) on the whole, the proof contains \(2 * log(n)\) elliptic curve points and one number (for pedants: one field element). verifying the proof takes logarithmic time in every step except one: computing the new \(g'_i\) values. this step is, unfortunately, linear. see also: dankrad feist's more detailed explanation of inner product arguments. extension to polynomial evaluations we can extend to polynomial evaluations with a simple clever trick. suppose we are trying to prove \(p(z) = a\). the prover and the verifier can extend the base points \(g_0 ... g_{n-1}\) by attaching powers of \(z\) to them: the new base points become \((g_0, 1), (g_1, z) ... (g_{n-1}, z^{n-1})\). these pairs can be treated as mathematical objects (for pedants: group elements) much like elliptic curve points themselves; to add them you do so element-by-element: \((a, x) + (b, y) =\) \((a + b,\ x + y)\), using elliptic curve addition for the points and regular field addition for the numbers. we can make a pedersen commitment using this extended base! now, here's a puzzle. suppose \(p(x) = \sum_i c_i x^i\), where \(p(z) = a\), would have a commitment \(c = \sum_i c_i g_i\) if we were to use the regular elliptic curve points we used before as a base. if we use the pairs \((g_i, z^i)\) as a base instead, the commitment would be \((c, y)\) for some \(y\). what must be the value of \(y\)? the answer is: it must be equal to \(a\)! this is easy to see: the commitment is \((c, y) = \sum_i c_i (g_i, z^i)\), which we can decompose as \((\sum_i c_i g_i,\ \sum_i c_i z^i)\). the former is equal to \(c\), and the latter is just the evaluation \(p(z)\)! hence, if \(c\) is a "regular" commitment to \(p\) using \(\{g_0 ... g_{n-1}\}\) as a base, then to prove that \(p(z) = a\) we need only use the same protocol above, but proving that \((c, a)\) is a valid commitment using \((g_0, 1), (g_1, z) ... (g_{n-1}, z^{n-1})\) as a base! note that in practice, this is usually done slightly differently as an optimization: instead of attaching the numbers to the points and explicitly dealing with structures of the form \((g_i, z^i)\), we add another randomly chosen base point \(h\) and express it as \(g_i + z^i h\). this saves space. see here for an example implementation of this whole protocol. so, how do we combine these proofs? suppose that you are given two polynomial evaluation proofs, with different polynomials and different evaluation points, and want to make a proof that they are both correct. you have: proof \(\pi_1\) proving that \(p_1(z_1) = y_1\), where \(p_1\) is represented by \(com(p_1) = c_1\) proof \(\pi_2\) proving that \(p_2(z_2) = y_2\), where \(p_2\) is represented by \(com(p_2) = c_2\) verifying each proof takes linear time. we want to make a proof that proves that both proofs are correct. this will still take linear time, but the verifier will only have to make one round of linear time verification instead of two. we start off with an observation. the only linear-time step in performing the verification of the proofs is computing the \(g'_i\) values. this is \(o(n)\) work because you have to combine \(\frac{n}{2}\) pairs of \(g_i\) values into \(g'_i\) values, then \(\frac{n}{4}\) pairs of \(g'_i\) values into \(g''_i\) values, and so on, for a total of \(n\) combinations of pairs. but if you look at the algorithm carefully, you will notice that we don't actually need any of the intermediate \(g'_i\) values; we only need the final \(g^*_0\). this \(g^*_0\) is a linear combination of the initial \(g_i\) values. what are the coefficients to that linear combination? it turns out that the \(g_i\) coefficient is the \(x^i\) term of this polynomial: \[(x + \alpha_1) * (x^2 + \alpha_2)\ *\ ...\ *\ (x^{\frac{n}{2}} + \alpha_{log(n)}) \] this is using the \(c' = \alpha c + \alpha^2 l + r\) version we mentioned above. the ability to directly compute \(g^*_0\) as a linear combination already cuts down our work to \(o(\frac{n}{log(n)})\) due to fast linear combination algorithms, but we can go further. the above polynomial has degree \(n 1\), with \(n\) nonzero coefficients. but its un-expanded form has size \(log(n)\), and so you can evaluate the polynomial at any point in \(o(log(n))\) time. additionally, you might notice that \(g^*_0\) is a commitment to this polynomial, so we can directly prove evaluations! so here is what we do: the prover computes the above polynomial for each proof; we'll call these polynomials \(k_1\) with \(com(k_1) = d_1\) and \(k_2\) with \(com(k_2) = d_2\). in a "normal" verification, the verifier would be computing \(d_1\) and \(d_2\) themselves as these are just the \(g^*_0\) values for their respective proofs. here, the prover provides \(d_1\) and \(d_2\) and the rest of the work is proving that they're correct. to prove the correctness of \(d_1\) and \(d_2\) we'll prove that they're correct at a random point. we choose a random point \(t\), and evaluate both \(e_1 = k_1(t)\) and \(e_2 = k_2(t)\) the prover generates a random linear combination \(l(x) = k_1(x) + rk_2(x)\) (and the verifier can generate \(com(l) = d_1 + rd_2\)). the prover now just needs to make a single proof that \(l(t) = e_1 + re_2\). the verifier still needs to do a bunch of extra steps, but all of those steps take either \(o(1)\) or \(o(log(n))\) work: evaluate \(e_1 = k_1(t)\) and \(e_2 = k_2(t)\), calculate the \(\alpha_i\) coefficients of both \(k_i\) polynomials in the first place, do the elliptic curve addition \(com(l) = d_1 + rd_2\). but this all takes vastly less than linear time, so all in all we still benefit: the verifier only needs to do the linear-time step of computing a \(g^*_0\) point themselves once. this technique can easily be generalized to merge \(m > 2\) signatures. from merging ipas to merging ipa-based snarks: halo now, we get into the core mechanic of the halo protocol being integrated in zcash, which uses this proof combining technique to create a recursive proof system. the setup is simple: suppose you have a chain, where each block has an associated ipa-based snark (see here for how generic snarks from polynomial commitments work) proving its correctness. you want to create a new block, building on top of the previous tip of the chain. the new block should have its own ipa-based snark proving the correctness of the block. in fact, this proof should cover both the correctness of the new block and the correctness of the previous block's proof of the correctness of the entire chain before it. ipa-based proofs by themselves cannot do this, because a proof of a statement takes longer to verify than checking the statement itself, so a proof of a proof will take even longer to verify than both proofs separately. but proof merging can do it! essentially, we use the usual "recursive snark" technique to verify the proofs, except the "proof of a proof" part is only proving the logarithmic part of the work. we add an extra chain of aggregate proofs, using a trick similar to the proof merging scheme above, to handle the linear part of the work. to verify the whole chain, the verifier need only verify one linear-time proof at the very tip of the chain. the precise details are somewhat different from the exact proof-combining trick in the previous section for efficiency reasons. instead of using the proof-combining trick to combine multiple proofs, we use it on a single proof, just to re-randomize the point that the polynomial committed to by \(g^*_0\) needs to be evaluated at. we then use the same newly chosen evaluation point to evaluate the polynomials in the proof of the block's correctness, which allows us to prove the polynomial evaluations together in a single ipa. expressed in math: let \(p(z) = a\) be the previous statement that needs to be proven the prover generates \(g^*_0\) the prover proves the correctness of the new block plus the logarithmic work in the previous statements by generating a plonk proof: \(q_l * a + q_r * b + q_o * c + q_m * a * b + q_c = z * h\) the prover chooses a random point \(t\), and proves the evaluation of a linear combination of \(\{g^*_0,\ q_l,\ a,\ q_r,\ b,\ q_o,\ c,\ q_m,\ q_c,\ z,\ h\}\) at \(t\). we can then check the above equation, replacing each polynomial with its now-verified evaluation at \(t\), to verify the plonk proof. incremental verification, more generally the size of each "step" does not need to be a full block verification; it could be something as small as a single step of a virtual machine. the smaller the steps the better: it ensures that the linear work that the verifier ultimately has to do at the end is less. the only lower bound is that each step has to be big enough to contain a snark verifying the \(log(n)\) portion of the work of a step. but regardless of the fine details, this mechanism allows us to make succinct and easy-to-verify snarks, including easy support for recursive proofs that allow you to extend proofs in real time as the computation extends and even have different provers to do different parts of the proving work, all without pairings or a trusted setup! the main downside is some extra technical complexity, compared with a "simple" polynomial-based proof using eg. kzg-based commitments. technology cryptographic assumptions proof size verification time fri hashes only (quantum safe!) large (10-200 kb) medium (poly-logarithmic) inner product arguments (ipas) basic elliptic curves medium (1-3 kb) very high (linear) kzg commitments elliptic curves + pairings + trusted setup short (~500 bytes) low (constant) ipa + halo-style aggregation basic elliptic curves medium (1-3 kb) medium (constant but higher than kzg) not just polynomials! merging r1cs proofs a common alternative to building snarks out of polynomial games is building snarks out of matrix-vector multiplication games. polynomials and vectors+matrices are both natural bases for snark protocols because they are mathematical abstractions that can store and compute over large amounts of data at the same time, and that admit commitment schemes that allow verifiers to check equations quickly. in r1cs (see a more detailed description here), an instance of the game consists of three matrices \(a\), \(b\), \(c\), and a solution is a vector \(z\) such that \((a \cdot z) \circ (b \cdot z) = c \cdot z\) (the problem is often in practice restricted further by requiring the prover to make part of \(z\) public and requiring the last entry of \(z\) to be 1). an r1cs instance with a single constraint (so \(a\), \(b\) and \(c\) have width 1), with a satisfying \(z\) vector, though notice that here the \(z\) appears on the left and has 1 in the top position instead of the bottom. just like with polynomial-based snarks, this r1cs game can be turned into a proof scheme by creating commitments to \(a\), \(b\) and \(c\), requiring the prover to provide a commitment to (the private portion of) \(z\), and using fancy proving tricks to prove the equation \((a \cdot z) \circ (b \cdot z) = c \cdot z\), where \(\circ\) is item-by-item multiplication, without fully revealing any of these objects. and just like with ipas, this r1cs game has a proof merging scheme! ioanna tzialla et al describe such a scheme in a recent paper (see page 8-9 for their description). they first modify the game by introducing an expanded equation: \[ (a \cdot z) \circ (b \cdot z) u * (c \cdot z) = e\] for a "base" instance, \(u = 1\) and \(e = 0\), so we get back the original r1cs equation. the extra slack variables are added to make aggregation possible; aggregated instances will have other values of \(u\) and \(e\). now, suppose that you have two solutions to the same instance, though with different \(u\) and \(e\) variables: \[(a \cdot z_1) \circ (b \cdot z_1) u_1 * (c \cdot z_1) = e_1\] \[(a \cdot z_2) \circ (b \cdot z_2) u_2 * (c \cdot z_2) = e_2\] the trick involves taking a random linear combination \(z_3 = z_1 + r z_2\), and making the equation work with this new value. first, let's evaluate the left side: \[ (a \cdot (z_1 + rz_2)) \circ (b \cdot (z_1 + rz_2)) (u_1 + ru_2)*(c \cdot (z_1 + rz_2)) \] this expands into the following (grouping the \(1\), \(r\) and \(r^2\) terms together): \[(a \cdot z_1) \circ (b \cdot z_1) u_1 * (c \cdot z_1)\] \[r((a \cdot z_1) \circ (b \cdot z_2) + (a \cdot z_2) \circ (b \cdot z_1) u_1 * (c \cdot z_2) u_2 * (c \cdot z_1))\] \[r^2((a \cdot z_2) \circ (b \cdot z_2) u_2 * (c \cdot z_2))\] the first term is just \(e_1\); the third term is \(r^2 * e_2\). the middle term is very similar to the cross-term (the yellow + green areas) near the very start of this post. the prover simply provides the middle term (without the \(r\) factor), and just like in the ipa proof, the randomization forces the prover to be honest. hence, it's possible to make merging schemes for r1cs-based protocols too. interestingly enough, we don't even technically need to have a "succinct" protocol for proving the \[ (a \cdot z) \circ (b \cdot z) = u * (c \cdot z) + e\] relation at the end; instead, the prover could just prove by opening all the commitments directly! this would still be "succinct" because the verifier would only need to verify one proof that actually represents an arbitrarily large number of statements. however, in practice having a succinct protocol for this last step is better because it keeps the proofs smaller, and tzialla et al's paper provides such a protocol too (see page 10). recap we don't know of a way to make a commitment to a size-\(n\) polynomial where evaluations of the polynomial can be verified in \(< o(n)\) time directly. the best that we can do is make a \(log(n)\) sized proof, where all of the work to verify it is logarithmic except for one final \(o(n)\)-time piece. but what we can do is merge multiple proofs together. given \(m\) proofs of evaluations of size-\(n\) polynomials, you can make a proof that covers all of these evaluations, that takes logarithmic work plus a single size-\(n\) polynomial proof to verify. with some clever trickery, separating out the logarithmic parts from the linear parts of proof verification, we can leverage this to make recursive snarks. these recursive snarks are actually more efficient than doing recursive snarks "directly"! in fact, even in contexts where direct recursive snarks are possible (eg. proofs with kzg commitments), halo-style techniques are typically used instead because they are more efficient. it's not just about polynomials; other games used in snarks like r1cs can also be aggregated in similar clever ways. no pairings or trusted setups required! the march toward faster and more efficient and safer zk-snarks just keeps going... frontier is coming what to expect, and how to prepare | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search frontier is coming what to expect, and how to prepare posted by stephan tual on july 22, 2015 protocol announcements we are only days away from launching 'frontier', the first milestone in the release of the ethereum project. frontier will be followed by 'homestead', 'metropolis' and 'serenity' throughout the coming year, each adding new features and improving the user friendliness and security of the platform. what is frontier? frontier is a live, but barebone implementation of the ethereum project. it's intended for technical users, specifically developers. during the frontier release, we expect early adopters and application developers to establish communities and start forming a live ecosystem. like their counterparts during the american frontier, these settlers will be presented with vast opportunities, but will also face many dangers. if building from source and command lines interfaces aren't your cup of tea, we strongly encourage you to wait for a more user-friendly release of the ethereum software before diving in. when is frontier going live? frontier is now feature complete and its codebase has been frozen for two weeks. as a team we are currently reviewing the final steps for the release, not all of them technical. there will be no countdown ethereum is not something that's centrally 'launched', but instead emerges from consensus. users will have to voluntarily download and run a specific version of the software, then generate and load the genesis block to join the official project's network. details on this process will be released soon. what can we expect in frontier? initial 'thawing': gas limits during the first few days the first software release of frontier will have a hardcoded gas limit per block of 5,000 gas. unlike the normal gas per block parameter, this special limit will not grow proportionally to the network usage effectively preventing transacting during the first few days. this 'thawing' period will enable miners to start their operations and early adopters to install their clients without having to 'rush'. after a few days (likely 3-4, but this could change), we'll release a small software update which all clients will have to install. this update will see the gas limit per block raised to 3 million, an initial number from which it will expand or contract as per the default miner settings. bugs, issues and complications we're very happy with how the 'olympic' testing phase of the ethereum testnet took shape. that said, the work on the frontier software is far from over. expect weekly updates which will give you access to better, more stable clients. many of the planned frontier gotchas (which included a chain reset at homestead, limiting mining rewards to 10%, and centralized checkpointing) were deemed unnecessary. however, there are still big differences between frontier and homestead. in frontier, we're going to have issues, we're going to have updates, and there will be bugs users are taking their chances when using the software. there will be big (big) warning messages before developers are able to even install it. in frontier, documentation is limited, and the tools provided require advanced technical skills. the canary contracts the canary contracts are simple switches holding a value equal to 0 or 1. each contract is controlled by a different member of the eth/dev team and will be updated to ‘1’ if the internal frontier disaster recovery team flags up a consensus issue, such as a fork. within each frontier client, a check is made after each block against 4 contracts. if two out of four of these contracts have a value switched from 0 to 1, mining stops and a message urging the user to update their client is displayed. this is to prevent "fire and forget" miners from preventing a chain upgrade. this process is centralized and will only run for the duration of frontier. it helps preventing the risk of a prolonged period (24h+) of outage. stats, status and badblock websites you probably are already familiar with our network stats monitor, https://stats.ethdev.com/. it gives a quick overview of the health of the network, block resolution time and gas statistics. if you’d like to explore it further, i’ve made a brief video explaining the various kpis. remember that participation in the stats page is voluntary, and nodes have to add themselves before they appear on the panel. in addition to the stats page, we will have a status page at https://status.ethdev.com/ (no link as the site is not live yet) which will gives a concise overview of any issue that might be affecting frontier. use it as your first port of call if you think something might not be right. finally, if any of the clients receive an invalid block, they will refuse to process it send it to the bad block website (aka ‘sentinel’). this could mean a bug, or something more serious, such as a fork. either way, this process will alert our developers to potential issues on the network. the website itself is public and available at https://badblocks.ethdev.com (currently operating on the testnet). a clean testnet during the last couple of months, the ethereum test network was pushed to its limits in order to test scalability and block propagation times. as part of this test we encouraged users to spam the network with transactions, contract creation code and call to contracts, at times reaching over 25 transactions per second. this has led the test network chain to grow to a rather unwieldy size, making it difficult for new users to catch up. for this reason, and shortly after the frontier release, there will be a new test network following the same rules as frontier. olympic rewards distribution during the olympic phase there were a number of rewards for various achievements including mining prowess. a large number of you participated and earned rewards a special mention goes to phistr90, dino and samuel lavery for their help during the stress tests. note that rewards will not be part of the frontier genesis block, but instead will be handed out by a foundation bot during the weeks following the release. how do i get started with frontier? the tools frontier and all its dependencies will be made available as a single line installer on our website at https://www.ethereum.org/. a single line installer will be provided for osx, linux and windows. of course, more advanced users will still be able to install everything from source, or use a binary build from our automated build bots. once frontier has been installed on their machines, users will need to generate the genesis block themselves, then load it into their frontier clients. a script and instructions on how to do this will be provided as part of the new ethereum website, as well as our various wikis. we’re often asked how existing users will switch from the test network to the live network: it will be done through a switch at the geth console (--networkid). by default the new build will aim to connect to the live network, to switch back to the test network you’ll simply indicate a network id of ‘0’. the documentation to get started with ethereum, the best place is our official gitbook. after consulting the gitbook, you can dig into the official solidity tutorial. for more in-depth information, please consult the main wiki, go client wiki and c++ client wiki. finally, if it's mining you'd like to learn more about, a mining faq and guide are regularly updated on our forums. getting help ethereum is an open source software project and as such, all help is provided through the community channels. if you face issues, the first port of call should be our forums, followed by our ethereum chat channels. if you're on the other hand are experiencing problems specific to your ether sale wallet, such as not being able to load your pre sale purchase, the helpdesk address will continue operating throughout frontier (and probably beyond). and of course, you can also find help locally at one of our 115 meetups around the world if your city isn’t listed, we encourage you to create one. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements security alert – [implementation bug in go clients causing increase in difficulty – fixed – miners check and update go clients] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security alert – [implementation bug in go clients causing increase in difficulty – fixed – miners check and update go clients] posted by jutta steiner on september 3, 2015 security implementation bug in the go client leads to steady increase of difficulty independent of hashing power. affected configurations: all go client versions v1.0.x, v1.1.x, release and develop branches. the bug was introduced in a recent update and release through commit https://github.com/ethereum/go-ethereum/commit/7324176f702a77fc331bf16a968d2eb4bccce021 which went into the affected client versions. all miners running earlier mentioned versions are affected and are advised to update as soon as possible. likelihood: high severity: medium impact: increase in block time will lead to an exponential increase in difficulty details: a bug in the go client leads to steady increase in difficulty in the following block, as timestamp in new block = timestamp + 1 of old block, regardless of the actual time, when mining. this leads to an increase in the difficulty independently of the hashing power effects on expected chain reorganisation depth: none proposed temporary workaround: none remedial action taken by ethereum: provision of hotfixes as below: if using the ppa: sudo apt-get update then sudo apt-get upgrade if using brew: brew update then brew reinstall ethereum if using a windows binary: download the updated binary from the release page if you are building from source: git pull followed by make geth (please use the master branch 587669215b878566c4a7b91fbf88a6fd2ec4f46a) previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements conference, alpha testnet and ether pre-sale updates | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search conference, alpha testnet and ether pre-sale updates posted by vitalik buterin on january 29, 2014 research & development important notice: any information from this post regarding the ether sale is highly outdated and probably inaccurate. please only consult the latest blog posts and official materials at ethereum.org for information on the sale ethereum received an incredible response at the miami bitcoin conference. we traveled there anticipating many technical questions as well as a philosophical discussion about the purpose of ethereum; however, the overwhelming amount of interest and enthusiasm for the project was much larger than we had anticipated. vitalik’s presentation was met with both a standing ovation and a question queue that took hours to address. because we intend on providing an equal opportunity to all those who want to be involved, and are reviewing the relevant logistical and regulatory issues for a token sale of this scale, we have decided to postpone the feb 1 launch of the sale. we will make the announcement of the new sale launch date on our official website: ethereum.org. the ethereum project is also excited to announce the alpha release of the open source testnet client to the community at the beginning of february. this will give people an opportunity to get involved with the project and experiment with ethereum scripts and contracts, and gain a better understanding of the technical properties of the ethereum platform. launching the testnet at this date will give those interested in the fundraiser a chance to better understand what the ethereum project is about before participating. the testnet will include full support for sending and receiving transactions, the initial version of the scripting language as described in our whitepaper, and may or may not include mining. this will also be the first major cryptocurrency project to have two official clients released at the same time, with one written in c++ and the other in go; a python client is also in the works. a compiler from the ethereum cll to ethereum script will be released very soon. a note on security the ether sale will not be launching on february 1st and any attempt to collect funds at this time should be considered a scam. there have been some scams in the forums thus please use caution and only consider information posted on ethereum.org to be legitimate. it is important to reinforce to all that only information released and posted at ethereum.org should be trusted as many are likely to impersonate us. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle in defense of bitcoin maximalism 2022 apr 01 see all posts we've been hearing for years that the future is blockchain, not bitcoin. the future of the world won't be one major cryptocurrency, or even a few, but many cryptocurrencies and the winning ones will have strong leadership under one central roof to adapt rapidly to users' needs for scale. bitcoin is a boomer coin, and ethereum is soon to follow; it will be newer and more energetic assets that attract the new waves of mass users who don't care about weird libertarian ideology or "self-sovereign verification", are turned off by toxicity and anti-government mentality, and just want blockchain defi and games that are fast and work. but what if this narrative is all wrong, and the ideas, habits and practices of bitcoin maximalism are in fact pretty close to correct? what if bitcoin is far more than an outdated pet rock tied to a network effect? what if bitcoin maximalists actually deeply understand that they are operating in a very hostile and uncertain world where there are things that need to be fought for, and their actions, personalities and opinions on protocol design deeply reflect that fact? what if we live in a world of honest cryptocurrencies (of which there are very few) and grifter cryptocurrencies (of which there are very many), and a healthy dose of intolerance is in fact necessary to prevent the former from sliding into the latter? that is the argument that this post will make. we live in a dangerous world, and protecting freedom is serious business hopefully, this is much more obvious now than it was six weeks ago, when many people still seriously thought that vladimir putin is a misunderstood and kindly character who is merely trying to protect russia and save western civilization from the gaypocalypse. but it's still worth repeating. we live in a dangerous world, where there are plenty of bad-faith actors who do not listen to compassion and reason. a blockchain is at its core a security technology a technology that is fundamentally all about protecting people and helping them survive in such an unfriendly world. it is, like the phial of galadriel, "a light to you in dark places, when all other lights go out". it is not a low-cost light, or a fluorescent hippie energy-efficient light, or a high-performance light. it is a light that sacrifices on all of those dimensions to optimize for one thing and one thing only: to be a light that does when it needs to do when you're facing the toughest challenge of your life and there is a friggin twenty foot spider staring at you in the face. source: https://www.blackgate.com/2014/12/23/frodo-baggins-lady-galadriel-and-the-games-of-the-mighty/ blockchains are being used every day by unbanked and underbanked people, by activists, by sex workers, by refugees, and by many other groups either who are uninteresting for profit-seeking centralized financial institutions to serve, or who have enemies that don't want them to be served. they are used as a primary lifeline by many people to make their payments and store their savings. and to that end, public blockchains sacrifice a lot for security: blockchains require each transaction to be independently verified thousands of times to be accepted. unlike centralized systems that confirm transactions in a few hundred milliseconds, blockchains require users to wait anywhere from 10 seconds to 10 minutes to get a confirmation. blockchains require users to be fully in charge of authenticating themselves: if you lose your key, you lose your coins. blockchains sacrifice privacy, requiring even crazier and more expensive technology to get that privacy back. what are all of these sacrifices for? to create a system that can survive in an unfriendly world, and actually do the job of being "a light in dark places, when all other lights go out". excellent at that task requires two key ingredients: (i) a robust and defensible technology stack and (ii) a robust and defensible culture. the key property to have in a robust and defensible technology stack is a focus on simplicity and deep mathematical purity: a 1 mb block size, a 21 million coin limit, and a simple nakamoto consensus proof of work mechanism that even a high school student can understand. the protocol design must be easy to justify decades and centuries down the line; the technology and parameter choices must be a work of art. the second ingredient is the culture of uncompromising, steadfast minimalism. this must be a culture that can stand unyieldingly in defending itself against corporate and government actors trying to co-opt the ecosystem from outside, as well as bad actors inside the crypto space trying to exploit it for personal profit, of which there are many. now, what do bitcoin and ethereum culture actually look like? well, let's ask kevin pham: don't believe this is representative? well, let's ask kevin pham again: now, you might say, this is just ethereum people having fun, and at the end of the day they understand what they have to do and what they are dealing with. but do they? let's look at the kinds of people that vitalik buterin, the founder of ethereum, hangs out with: vitalik hangs out with elite tech ceos in beijing, china. vitalik meets vladimir putin in russia. vitalik meets nir bakrat, mayor of jerusalem. vitalik shakes hands with argentinian former president mauricio macri. vitalik gives a friendly hello to eric schmidt, former ceo of google and advisor to us department of defense. vitalik has his first of many meetings with audrey tang, digital minister of taiwan. and this is only a small selection. the immediate question that anyone looking at this should ask is: what the hell is the point of publicly meeting with all these people? some of these people are very decent entrepreneurs and politicians, but others are actively involved in serious human rights abuses that vitalik certainly does not support. does vitalik not realize just how much some of these people are geopolitically at each other's throats? now, maybe he is just an idealistic person who believes in talking to people to help bring about world peace, and a follower of frederick douglass's dictum to "unite with anybody to do right and with nobody to do wrong". but there's also a simpler hypothesis: vitalik is a hippy-happy globetrotting pleasure and status-seeker, and he deeply enjoys meeting and feeling respected by people who are important. and it's not just vitalik; companies like consensys are totally happy to partner with saudi arabia, and the ecosystem as a whole keeps trying to look to mainstream figures for validation. now ask yourself the question: when the time comes, actually important things are happening on the blockchain actually important things that offend people who are powerful which ecosystem would be more willing to put its foot down and refuse to censor them no matter how much pressure is applied on them to do so? the ecosystem with globe-trotting nomads who really really care about being everyone's friend, or the ecosystem with people who take pictures of themslves with an ar15 and an axe as a side hobby? currency is not "just the first app". it's by far the most successful one. many people of the "blockchain, not bitcoin" persuasion argue that cryptocurrency is the first application of blockchains, but it's a very boring one, and the true potential of blockchains lies in bigger and more exciting things. let's go through the list of applications in the ethereum whitepaper: issuing tokens financial derivatives stablecoins identity and reputation systems decentralized file storage decentralized autonomous organizations (daos) peer-to-peer gambling prediction markets many of these categories have applications that have launched and that have at least some users. that said, cryptocurrency people really value empowering under-banked people in the "global south". which of these applications actually have lots of users in the global south? as it turns out, by far the most successful one is storing wealth and payments. 3% of argentinians own cryptocurrency, as do 6% of nigerians and 12% of people in ukraine. by far the biggest instance of a government using blockchains to accomplish something useful today is cryptocurrency donations to the government of ukraine, which have raised more than $100 million if you include donations to non-governmental ukraine-related efforts. what other application has anywhere close to that level of actual, real adoption today? perhaps the closest is ens. daos are real and growing, but today far too many of them are appealing to wealthy rich-country people whose main interest is having fun and using cartoon-character profiles to satisfy their first-world need for self-expression, and not build schools and hospitals and solve other real world problems. thus, we can see the two sides pretty clearly: team "blockchain", privileged people in wealthy countries who love to virtue-signal about "moving beyond money and capitalism" and can't help being excited about "decentralized governance experimentation" as a hobby, and team "bitcoin", a highly diverse group of both rich and poor people in many countries around the world including the global south, who are actually using the capitalist tool of free self-sovereign money to provide real value to human beings today. focusing exclusively on being money makes for better money a common misconception about why bitcoin does not support "richly stateful" smart contracts goes as follows. bitcoin really really values being simple, and particularly having low technical complexity, to reduce the chance that something will go wrong. as a result, it doesn't want to add the more complicated features and opcodes that are necessary to be able to support more complicated smart contracts in ethereum. this misconception is, of course, wrong. in fact, there are plenty of ways to add rich statefulness into bitcoin; search for the word "covenants" in bitcoin chat archives to see many proposals being discussed. and many of these proposals are surprisingly simple. the reason why covenants have not been added is not that bitcoin developers see the value in rich statefulness but find even a little bit more protocol complexity intolerable. rather, it's because bitcoin developers are worried about the risks of the systemic complexity that rich statefulness being possible would introduce into the ecosystem! a recent paper by bitcoin researchers describes some ways to introduce covenants to add some degree of rich statefulness to bitcoin. ethereum's battle with miner-extractable value (mev) is an excellent example of this problem appearing in practice. it's very easy in ethereum to build applications where the next person to interact with some contract gets a substantial reward, causing transactors and miners to fight over it, and contributing greatly to network centralization risk and requiring complicated workarounds. in bitcoin, building such systemically risky applications is hard, in large part because bitcoin lacks rich statefulness and focuses on the simple (and mev-free) use case of just being money. systemic contagion can happen in non-technical ways too. bitcoin just being money means that bitcoin requires relatively few developers, helping to reduce the risk that developers will start demanding to print themselves free money to build new protocol features. bitcoin just being money reduces pressure for core developers to keep adding features to "keep up with the competition" and "serve developers' needs". in so many ways, systemic effects are real, and it's just not possible for a currency to "enable" an ecosystem of highly complex and risky decentralized applications without that complexity biting it back somehow. bitcoin makes the safe choice. if ethereum continues its layer-2-centric approach, eth-the-currency may gain some distance from the application ecosystem that it's enabling and thereby get some protection. so-called high-performance layer-1 platforms, on the other hand, stand no chance. in general, the earliest projects in an industry are the most "genuine" many industries and fields follow a similar pattern. first, some new exciting technology either gets invented, or gets a big leap of improvement to the point where it's actually usable for something. at the beginning, the technology is still clunky, it is too risky for almost anyone to touch as an investment, and there is no "social proof" that people can use it to become successful. as a result, the first people involved are going to be the idealists, tech geeks and others who are genuinely excited about the technology and its potential to improve society. once the technology proves itself enough, however, the normies come in an event that in internet culture is often called eternal september. and these are not just regular kindly normies who want to feel part of something exciting, but business normies, wearing suits, who start scouring the ecosystem wolf-eyed for ways to make money with armies of venture capitalists just as eager to make their own money supporting them from the sidelines. in the extreme cases, outright grifters come in, creating blockchains with no redeeming social or technical value which are basically borderline scams. but the reality is that the line from "altruistic idealist" and "grifter" is really a spectrum. and the longer an ecosystem keeps going, the harder it is for any new project on the altruistic side of the spectrum to get going. one noisy proxy for the blockchain industry's slow replacement of philosophical and idealistic values with short-term profit-seeking values is the larger and larger size of premines: the allocations that developers of a cryptocurrency give to themselves. source for insider allocations: messari. which blockchain communities deeply value self-sovereignty, privacy and decentralization, and are making to get big sacrifices to get it? and which blockchain communities are just trying to pump up their market caps and make money for founders and investors? the above chart should make it pretty clear. intolerance is good the above makes it clear why bitcoin's status as the first cryptocurrency gives it unique advantages that are extremely difficult for any cryptocurrency created within the last five years to replicate. but now we get to the biggest objection against bitcoin maximalist culture: why is it so toxic? the case for bitcoin toxicity stems from conquest's second law. in robert conquest's original formulation, the law says that "any organization not explicitly and constitutionally right-wing will sooner or later become left-wing". but really, this is just a special case of a much more general pattern, and one that in the modern age of relentlessly homogenizing and conformist social media is more relevant than ever: if you want to retain an identity that is different from the mainstream, then you need a really strong culture that actively resists and fights assimilation into the mainstream every time it tries to assert its hegemony. blockchains are, as i mentioned above, very fundamentally and explicitly a counterculture movement that is trying to create and preserve something different from the mainstream. at a time when the world is splitting up into great power blocs that actively suppress social and economic interaction between them, blockchains are one of the very few things that can remain global. at a time when more and more people are reaching for censorship to defeat their short-term enemies, blockchains steadfastly continue to censor nothing. the only correct way to respond to "reasonable adults" trying to tell you that to "become mainstream" you have to compromise on your "extreme" values. because once you compromise once, you can't stop. blockchain communities also have to fight bad actors on the inside. bad actors include: scammers, who make and sell projects that are ultimately valueless (or worse, actively harmful) but cling to the "crypto" and "decentralization" brand (as well as highly abstract ideas of humanism and friendship) for legitimacy. collaborationists, who publicly and loudly virtue-signal about working together with governments and actively try to convince governments to use coercive force against their competitors. corporatists, who try to use their resources to take over the development of blockchains, and often push for protocol changes that enable centralization. one could stand against all of these actors with a smiling face, politely telling the world why they "disagree with their priorities". but this is unrealistic: the bad actors will try hard to embed themselves into your community, and at that point it becomes psychologically hard to criticize them with the sufficient level of scorn that they truly require: the people you're criticizing are friends of your friends. and so any culture that values agreeableness will simply fold before the challenge, and let scammers roam freely through the wallets of innocent newbies. what kind of culture won't fold? a culture that is willing and eager to tell both scammers on the inside and powerful opponents on the outside to go the way of the russian warship. weird crusades against seed oils are good one powerful bonding tool to help a community maintain internal cohesion around its distinctive values, and avoid falling into the morass that is the mainstream, is weird beliefs and crusades that are in a similar spirit, even if not directly related, to the core mission. ideally, these crusades should be at least partially correct, poking at a genuine blind spot or inconsistency of mainstream values. the bitcoin community is good at this. their most recent crusade is a war against seed oils, oils derived from vegetable seeds high in omega-6 fatty acids that are harmful to human health. this bitcoiner crusade gets treated skeptically when reviewed in the media, but the media treats the topic much more favorably when "respectable" tech firms are tackling it. the crusade helps to remind bitcoiners that the mainstream media is fundamentally tribal and hypocritical, and so the media's shrill attempts to slander cryptocurrency as being primarily for money laundering and terrorism should be treated with the same level of scorn. be a maximalist maximalism is often derided in the media as both a dangerous toxic right-wing cult, and as a paper tiger that will disappear as soon as some other cryptocurrency comes in and takes over bitcoin's supreme network effect. but the reality is that none of the arguments for maximalism that i describe above depend at all on network effects. network effects really are logarithmic, not quadratic: once a cryptocurrency is "big enough", it has enough liquidity to function and multi-cryptocurrency payment processors will easily add it to their collection. but the claim that bitcoin is an outdated pet rock and its value derives entirely from a walking-zombie network effect that just needs a little push to collapse is similarly completely wrong. crypto-assets like bitcoin have real cultural and structural advantages that make them powerful assets worth holding and using. bitcoin is an excellent example of the category, though it's certainly not the only one; other honorable cryptocurrencies do exist, and maximalists have been willing to support and use them. maximalism is not just bitcoin-for-the-sake-of-bitcoin; rather, it's a very genuine realization that most other cryptoassets are scams, and a culture of intolerance is unavoidable and necessary to protect newbies and make sure at least one corner of that space continues to be a corner worth living in. it's better to mislead ten newbies into avoiding an investment that turns out good than it is to allow a single newbie to get bankrupted by a grifter. it's better to make your protocol too simple and fail to serve ten low-value short-attention-span gambling applications than it is to make it too complex and fail to serve the central sound money use case that underpins everything else. and it's better to offend millions by standing aggressively for what you believe in than it is to try to keep everyone happy and end up standing for nothing. be brave. fight for your values. be a maximalist. dark mode toggle why sharding is great: demystifying the technical properties 2021 apr 07 see all posts special thanks to dankrad feist and aditya asgaonkar for review sharding is the future of ethereum scalability, and it will be key to helping the ecosystem support many thousands of transactions per second and allowing large portions of the world to regularly use the platform at an affordable cost. however, it is also one of the more misunderstood concepts in the ethereum ecosystem and in blockchain ecosystems more broadly. it refers to a very specific set of ideas with very specific properties, but it often gets conflated with techniques that have very different and often much weaker security properties. the purpose of this post will be to explain exactly what specific properties sharding provides, how it differs from other technologies that are not sharding, and what sacrifices a sharded system has to make to achieve these properties. one of the many depictions of a sharded version of ethereum. original diagram by hsiao-wei wang, design by quantstamp. the scalability trilemma the best way to describe sharding starts from the problem statement that shaped and inspired the solution: the scalability trilemma. the scalability trilemma says that there are three properties that a blockchain try to have, and that, if you stick to "simple" techniques, you can only get two of those three. the three properties are: scalability: the chain can process more transactions than a single regular node (think: a consumer laptop) can verify. decentralization: the chain can run without any trust dependencies on a small group of large centralized actors. this is typically interpreted to mean that there should not be any trust (or even honest-majority assumption) of a set of nodes that you cannot join with just a consumer laptop. security: the chain can resist a large percentage of participating nodes trying to attack it (ideally 50%; anything above 25% is fine, 5% is definitely not fine). now we can look at the three classes of "easy solutions" that only get two of the three: traditional blockchains including bitcoin, pre-pos/sharding ethereum, litecoin, and other similar chains. these rely on every participant running a full node that verifies every transaction, and so they have decentralization and security, but not scalability. high-tps chains including the dpos family but also many others. these rely on a small number of nodes (often 10-100) maintaining consensus among themselves, with users having to trust a majority of these nodes. this is scalable and secure (using the definitions above), but it is not decentralized. multi-chain ecosystems this refers to the general concept of "scaling out" by having different applications live on different chains and using cross-chain-communication protocols to talk between them. this is decentralized and scalable, but it is not secure, because an attacker need only get a consensus node majority in one of the many chains (so often <1% of the whole ecosystem) to break that chain and possibly cause ripple effects that cause great damage to applications in other chains. sharding is a technique that gets you all three. a sharded blockchain is: scalable: it can process far more transactions than a single node decentralized: it can survive entirely on consumer laptops, with no dependency on "supernodes" whatsoever secure: an attacker can't target a small part of the system with a small amount of resources; they can only try to dominate and attack the whole thing the rest of the post will be describing how sharded blockchains manage to do this. sharding through random sampling the easiest version of sharding to understand is sharding through random sampling. sharding through random sampling has weaker trust properties than the forms of sharding that we are building towards in the ethereum ecosystem, but it uses simpler technology. the core idea is as follows. suppose that you have a proof of stake chain with a large number (eg. 10000) validators, and you have a large number (eg. 100) blocks that need to be verified. no single computer is powerful enough to validate all of these blocks before the next set of blocks comes in. hence, what we do is we randomly split up the work of doing the verification. we randomly shuffle the validator list, and we assign the first 100 validators in the shuffled list to verify the first block, the second 100 validators in the shuffled list to verify the second block, etc. a randomly selected group of validators that gets assigned to verify a block (or perform some other task) is called a committee. when a validator verifies a block, they publish a signature attesting to the fact that they did so. everyone else, instead of verifying 100 entire blocks, now only verifies 10000 signatures a much smaller amount of work, especially with bls signature aggregation. instead of every block being broadcasted through the same p2p network, each block is broadcasted on a different sub-network, and nodes need only join the subnets corresponding to the blocks that they are responsible for (or are interested in for other reasons). consider what happens if each node's computing power increases by 2x. because each node can now safely validate 2x more signatures, you could cut the minimum staking deposit size to support 2x more validators, and so hence you can make 200 committees instead of 100. hence, you can verify 200 blocks per slot instead of 100. furthermore, each individual block could be 2x bigger. hence, you have 2x more blocks of 2x the size, or 4x more chain capacity altogether. we can introduce some math lingo to talk about what's going on. using big o notation, we use "o(c)" to refer to the computational capacity of a single node. a traditional blockchain can process blocks of size o(c). a sharded chain as described above can process o(c) blocks in parallel (remember, the cost to each node to verify each block indirectly is o(1) because each node only needs to verify a fixed number of signatures), and each block has o(c) capacity, and so the sharded chain's total capacity is o(c2). this is why we call this type of sharding quadratic sharding, and this effect is a key reason why we think that in the long run, sharding is the best way to scale a blockchain. frequently asked question: how is splitting into 100 committees different from splitting into 100 separate chains? there are two key differences: the random sampling prevents the attacker from concentrating their power on one shard. in a 100-chain multichain ecosystem, the attacker only needs ~0.5% of the total stake to wreak havoc: they can focus on 51% attacking a single chain. in a sharded blockchain, the attacker must have close to ~30-40% of the entire stake to do the same (in other words, the chain has shared security). certainly, they can wait until they get lucky and get 51% in a single shard by random chance despite having less than 50% of the total stake, but this gets exponentially harder for attackers that have much less than 51%. if an attacker has less than ~30%, it's virtually impossible. tight coupling: if even one shard gets a bad block, the entire chain reorgs to avoid it. there is a social contract (and in later sections of this document we describe some ways to enforce this technologically) that a chain with even one bad block in even one shard is not acceptable and should get thrown out as soon as it is discovered. this ensures that from the point of view of an application within the chain, there is perfect security: contract a can rely on contract b, because if contract b misbehaves due to an attack on the chain, that entire history reverts, including the transactions in contract a that misbehaved as a result of the malfunction in contract b. both of these differences ensure that sharding creates an environment for applications that preserves the key safety properties of a single-chain environment, in a way that multichain ecosystems fundamentally do not. improving sharding with better security models one common refrain in bitcoin circles, and one that i completely agree with, is that blockchains like bitcoin (or ethereum) do not completely rely on an honest majority assumption. if there is a 51% attack on such a blockchain, then the attacker can do some nasty things, like reverting or censoring transactions, but they cannot insert invalid transactions. and even if they do revert or censor transactions, users running regular nodes could easily detect that behavior, so if the community wishes to coordinate to resolve the attack with a fork that takes away the attacker's power they could do so quickly. the lack of this extra security is a key weakness of the more centralized high-tps chains. such chains do not, and cannot, have a culture of regular users running nodes, and so the major nodes and ecosystem players can much more easily get together and impose a protocol change that the community heavily dislikes. even worse, the users' nodes would by default accept it. after some time, users would notice, but by then the forced protocol change would be a fait accompli: the coordination burden would be on users to reject the change, and they would have to make the painful decision to revert a day's worth or more of activity that everyone had thought was already finalized. ideally, we want to have a form of sharding that avoids 51% trust assumptions for validity, and preserves the powerful bulwark of security that traditional blockchains get from full verification. and this is exactly what much of our research over the last few years has been about. scalable verification of computation we can break up the 51%-attack-proof scalable validation problem into two cases: validating computation: checking that some computation was done correctly, assuming you have possession of all the inputs to the computation validating data availability: checking that the inputs to the computation themselves are stored in some form where you can download them if you really need to; this checking should be performed without actually downloading the entire inputs themselves (because the data could be too large to download for every block) validating a block in a blockchain involves both computation and data availability checking: you need to be convinced that the transactions in the block are valid and that the new state root hash claimed in the block is the correct result of executing those transactions, but you also need to be convinced that enough data from the block was actually published so that users who download that data can compute the state and continue processing the blockchain. this second part is a very subtle but important concept called the data availability problem; more on this later. scalably validating computation is relatively easy; there are two families of techniques: fraud proofs and zk-snarks. fraud proofs are one way to verify computation scalably. the two technologies can be described simply as follows: fraud proofs are a system where to accept the result of a computation, you require someone with a staked deposit to sign a message of the form "i certify that if you make computation c with input x, you get output y". you trust these messages by default, but you leave open the opportunity for someone else with a staked deposit to make a challenge (a signed message saying "i disagree, the output is z"). only when there is a challenge, all nodes run the computation. whichever of the two parties was wrong loses their deposit, and all computations that depend on the result of that computation are recomputed. zk-snarks are a form of cryptographic proof that directly proves the claim "performing computation c on input x gives output y". the proof is cryptographically "sound": if c(x) does not equal y, it's computationally infeasible to make a valid proof. the proof is also quick to verify, even if running c itself takes a huge amount of time. see this post for more mathematical details on zk-snarks. computation based on fraud proofs is scalable because "in the normal case" you replace running a complex computation with verifying a single signature. there is the exceptional case, where you do have to verify the computation on-chain because there is a challenge, but the exceptional case is very rare because triggering it is very expensive (either the original claimer or the challenger loses a large deposit). zk-snarks are conceptually simpler they just replace a computation with a much cheaper proof verification but the math behind how they work is considerably more complex. there is a class of semi-scalable system which only scalably verifies computation, while still requiring every node to verify all the data. this can be made quite effective by using a set of compression tricks to replace most data with computation. this is the realm of rollups. scalable verification of data availability is harder a fraud proof cannot be used to verify availability of data. fraud proofs for computation rely on the fact that the inputs to the computation are published on-chain the moment the original claim is submitted, and so if someone challenges, the challenge execution is happening in the exact same "environment" that the original execution was happening. in the case of checking data availability, you cannot do this, because the problem is precisely the fact that there is too much data to check to publish it on chain. hence, a fraud proof scheme for data availability runs into a key problem: someone could claim "data x is available" without publishing it, wait to get challenged, and only then publish data x and make the challenger appear to the rest of the network to be incorrect. this is expanded on in the fisherman's dilemma: the core idea is that the two "worlds", one where v1 is an evil publisher and v2 is an honest challenger and the other where v1 is an honest publisher and v2 is an evil challenger, are indistinguishable to anyone who was not trying to download that particular piece of data at the time. and of course, in a scalable decentralized blockchain, each individual node can only hope to download a small portion of the data, so only a small portion of nodes would see anything about what went on except for the mere fact that there was a disagreement. the fact that it is impossible to distinguish who was right and who was wrong makes it impossible to have a working fraud proof scheme for data availability. frequently asked question: so what if some data is unavailable? with a zk-snark you can be sure everything is valid, and isn't that enough? unfortunately, mere validity is not sufficient to ensure a correctly running blockchain. this is because if the blockchain is valid but all the data is not available, then users have no way of updating the data that they need to generate proofs that any future block is valid. an attacker that generates a valid-but-unavailable block but then disappears can effectively stall the chain. someone could also withhold a specific user's account data until the user pays a ransom, so the problem is not purely a liveness issue. there are some strong information-theoretic arguments that this problem is fundamental, and there is no clever trick (eg. involving cryptographic accumulators) that can get around it. see this paper for details. so, how do you check that 1 mb of data is available without actually trying to download it? that sounds impossible! the key is a technology called data availability sampling. data availability sampling works as follows: use a tool called erasure coding to expand a piece of data with n chunks into a piece of data with 2n chunks such that any n of those chunks can recover the entire data. to check for availability, instead of trying to download the entire data, users simply randomly select a constant number of positions in the block (eg. 30 positions), and accept the block only when they have successfully found the chunks in the block at all of their selected positions. erasure codes transform a "check for 100% availability" (every single piece of data is available) problem into a "check for 50% availability" (at least half of the pieces are available) problem. random sampling solves the 50% availability problem. if less than 50% of the data is available, then at least one of the checks will almost certainly fail, and if at least 50% of the data is available then, while some nodes may fail to recognize a block as available, it takes only one honest node to run the erasure code reconstruction procedure to bring back the remaining 50% of the block. and so, instead of needing to download 1 mb to check the availability of a 1 mb block, you need only download a few kilobytes. this makes it feasible to run data availability checking on every block. see this post for how this checking can be efficiently implemented with peer-to-peer subnets. a zk-snark can be used to verify that the erasure coding on a piece of data was done correctly, and then merkle branches can be used to verify individual chunks. alternatively, you can use polynomial commitments (eg. kate (aka kzg) commitments), which essentially do erasure coding and proving individual elements and correctness verification all in one simple component and that's what ethereum sharding is using. recap: how are we ensuring everything is correct again? suppose that you have 100 blocks and you want to efficiently verify correctness for all of them without relying on committees. we need to do the following: each client performs data availability sampling on each block, verifying that the data in each block is available, while downloading only a few kilobytes per block even if the block as a whole is a megabyte or larger in size. a client only accepts a block when all data of their availability challenges have been correctly responded to. now that we have verified data availability, it becomes easier to verify correctness. there are two techniques: we can use fraud proofs: a few participants with staked deposits can sign off on each block's correctness. other nodes, called challengers (or fishermen) randomly check and attempt to fully process blocks. because we already checked data availability, it will always be possible to download the data and fully process any particular block. if they find an invalid block, they post a challenge that everyone verifies. if the block turns out to be bad, then that block and all future blocks that depend on that need to be re-computed. we can use zk-snarks. each block would come with a zk-snark proving correctness. in either of the above cases, each client only needs to do a small amount of verification work per block, no matter how big the block is. in the case of fraud proofs, occasionally blocks will need to be fully verified on-chain, but this should be extremely rare because triggering even one challenge is very expensive. and that's all there is to it! in the case of ethereum sharding, the near-term plan is to make sharded blocks data-only; that is, the shards are purely a "data availability engine", and it's the job of layer-2 rollups to use that secure data space, plus either fraud proofs or zk-snarks, to implement high-throughput secure transaction processing capabilities. however, it's completely possible to create such a built-in system to add "native" high-throughput execution. what are the key properties of sharded systems and what are the tradeoffs? the key goal of sharding is to come as close as possible to replicating the most important security properties of traditional (non-sharded) blockchains but without the need for each node to personally verify each transaction. sharding comes quite close. in a traditional blockchain: invalid blocks cannot get through because validating nodes notice that they are invalid and ignore them. unavailable blocks cannot get through because validating nodes fail to download them and ignore them. in a sharded blockchain with advanced security features: invalid blocks cannot get through because either: a fraud proof quickly catches them and informs the entire network of the block's incorrectness, and heavily penalizes the creator, or a zk-snark proves correctness, and you cannot make a valid zk-snark for an invalid block. unavailable blocks cannot get through because: if less than 50% of a block's data is available, at least one data availability sample check will almost certainly fail for each client, causing the client to reject the block, if at least 50% of a block's data is available, then actually the entire block is available, because it takes only a single honest node to reconstruct the rest of the block. traditional high-tps chains without sharding do not have a way of providing these guarantees. multichain ecosystems do not have a way of avoiding the problem of an attacker selecting one chain for attack and easily taking it over (the chains could share security, but if this was done poorly it would turn into a de-facto traditional high-tps chain with all its disadvantages, and if it was done well, it would just be a more complicated implementation of the above sharding techniques). sidechains are highly implementation-dependent, but they are typically vulnerable to either the weaknesses of traditional high-tps chains (this is if they share miners/validators), or the weaknesses of multichain ecosystems (this is if they do not share miners/validators). sharded chains avoid these issues. however, there are some chinks in the sharded system's armor. notably: sharded chains that rely only on committees are vulnerable to adaptive adversaries, and have weaker accountability. that is, if the adversary has the ability to hack into (or just shut down) any set of nodes of their choosing in real time, then they only need to attack a small number of nodes to break a single committee. furthermore, if an adversary (whether an adaptive adversary or just an attacker with 50% of the total stake) does break a single committee, only a few of their nodes (the ones in that committee) can be publicly confirmed to be participating in that attack, and so only a small amount of stake can be penalized. this is another key reason why data availability sampling together with either fraud proofs or zk-snarks are an important complement to random sampling techniques. data availability sampling is only secure if there is a sufficient number of online clients that they collectively make enough data availability sampling requests that the responses almost always overlap to comprise at least 50% of the block. in practice, this means that there must be a few hundred clients online (and this number increases the higher the ratio of the capacity of the system to the capacity of a single node). this is a few-of-n trust model generally quite trustworthy, but certainly not as robust as the 0-of-n trust that nodes in non-sharded chains have for availability. if the sharded chain relies on fraud proofs, then it relies on timing assumptions; if the network is too slow, nodes could accept a block as finalized before the fraud proof comes in showing that it is wrong. fortunately, if you follow a strict rule of reverting all invalid blocks once the invalidity is discovered, this threshold is a user-set parameter: each individual user chooses how long they wait until finality and if they didn't want long enough then suffer, but more careful users are safe. even still, this is a weakening of the user experience. using zk-snarks to verify validity solves this. there is a much larger amount of raw data that needs to be passed around, increasing the risk of failures under extreme networking conditions. small amounts of data are easier to send (and easier to safely hide, if a powerful government attempts to censor the chain) than larger amounts of data. block explorers need to store more data if they want to hold the entire chain. sharded blockchains depend on sharded peer-to-peer networks, and each individual p2p "subnet" is easier to attack because it has fewer nodes. the subnet model used for data availability sampling mitigates this because there is some redundancy between subnets, but even still there is a risk. these are valid concerns, though in our view they are far outweighed by the reduction in user-level centralization enabled by allowing more applications to run on-chain instead of through centralized layer-2 services. that said, these concerns, especially the last two, are in practice the real constraint on increasing a sharded chain's throughput beyond a certain point. there is a limit to the quadraticness of quadratic sharding. incidentally, the growing safety risks of sharded blockchains if their throughput becomes too high are also the key reason why the effort to extend to super-quadratic sharding has been largely abandoned; it looks like keeping quadratic sharding just quadratic really is the happy medium. why not centralized production and sharded verification? one alternative to sharding that gets often proposed is to have a chain that is structured like a centralized high-tps chain, except it uses data availability sampling and sharding on top to allow verification of validity and availability. this improves on centralized high-tps chains as they exist today, but it's still considerably weaker than a sharded system. this is for a few reasons: it's much harder to detect censorship by block producers in a high-tps chain. censorship detection requires either (i) being able to see every transaction and verify that there are no transactions that clearly deserve to get in that inexplicably fail to get in, or (ii) having a 1-of-n trust model in block producers and verifying that no blocks fail to get in. in a centralized high-tps chain, (i) is impossible, and (ii) is harder because the small node count makes even a 1-of-n trust model more likely to break, and if the chain has a block time that is too fast for das (as most centralized high-tps chains do), it's very hard to prove that a node's blocks are not being rejected simply because they are all being published too slowly. if a majority of block producers and ecosystem members tries to force through an unpopular protocol change, users' clients will certainly detect it, but it's much harder for the community to rebel and fork away because they would need to spin up a new set of very expensive high-throughput nodes to maintain a chain that keeps the old rules. centralized infrastructure is more vulnerable to censorship imposed by external actors. the high throughput of the block producing nodes makes them very detectable and easier to shut down. it's also politically and logistically easier to censor dedicated high-performance computation than it is to go after individual users' laptops. there's a stronger pressure for high-performance computation to move to centralized cloud services, increasing the risk that the entire chain will be run within 1-3 companies' cloud services, and hence risk of the chain going down because of many block producers failing simultaneously. a sharded chain with a culture of running validators on one's own hardware is again much less vulnerable to this. properly sharded systems are better as a base layer. given a sharded base layer, you can always create a centralized-production system (eg. because you want a high-throughput domain with synchronous composability for defi) layered on top by building it as a rollup. but if you have a base layer with a dependency on centralized block production, you cannot build a more-decentralized layer 2 on top. dark mode toggle how will ethereum's multi-client philosophy interact with zk-evms? 2023 mar 31 see all posts special thanks to justin drake for feedback and review one underdiscussed, but nevertheless very important, way in which ethereum maintains its security and decentralization is its multi-client philosophy. ethereum intentionally has no "reference client" that everyone runs by default: instead, there is a collaboratively-managed specification (these days written in the very human-readable but very slow python) and there are multiple teams making implementations of the spec (also called "clients"), which is what users actually run. each ethereum node runs a consensus client and an execution client. as of today, no consensus or execution client makes up more than 2/3 of the network. if a client with less than 1/3 share in its category has a bug, the network would simply continue as normal. if a client with between 1/3 and 2/3 share in its category (so, prysm, lighthouse or geth) has a bug, the chain would continue adding blocks, but it would stop finalizing blocks, giving time for developers to intervene. one underdiscussed, but nevertheless very important, major upcoming transition in the way the ethereum chain gets validated is the rise of zk-evms. snarks proving evm execution have been under development for years already, and the technology is actively being used by layer 2 protocols called zk rollups. some of these zk rollups are active on mainnet today, with more coming soon. but in the longer term, zk-evms are not just going to be for rollups; we want to use them to verify execution on layer 1 as well (see also: the verge). once that happens, zk-evms de-facto become a third type of ethereum client, just as important to the network's security as execution clients and consensus clients are today. and this naturally raises a question: how will zk-evms interact with the multi-client philosophy? one of the hard parts is already done: we already have multiple zk-evm implementations that are being actively developed. but other hard parts remain: how would we actually make a "multi-client" ecosystem for zk-proving correctness of ethereum blocks? this question opens up some interesting technical challenges and of course the looming question of whether or not the tradeoffs are worth it. what was the original motivation for ethereum's multi-client philosophy? ethereum's multi-client philosophy is a type of decentralization, and like decentralization in general, one can focus on either the technical benefits of architectural decentralization or the social benefits of political decentralization. ultimately, the multi-client philosophy was motivated by both and serves both. arguments for technical decentralization the main benefit of technical decentralization is simple: it reduces the risk that one bug in one piece of software leads to a catastrophic breakdown of the entire network. a historical situation that exemplifies this risk is the 2010 bitcoin overflow bug. at the time, the bitcoin client code did not check that the sum of the outputs of a transaction does not overflow (wrap around to zero by summing to above the maximum integer of \(2^{64} 1\)), and so someone made a transaction that did exactly that, giving themselves billions of bitcoins. the bug was discovered within hours, and a fix was rushed through and quickly deployed across the network, but had there been a mature ecosystem at the time, those coins would have been accepted by exchanges, bridges and other structures, and the attackers could have gotten away with a lot of money. if there had been five different bitcoin clients, it would have been very unlikely that all of them had the same bug, and so there would have been an immediate split, and the side of the split that was buggy would have probably lost. there is a tradeoff in using the multi-client approach to minimize the risk of catastrophic bugs: instead, you get consensus failure bugs. that is, if you have two clients, there is a risk that the clients have subtly different interpretations of some protocol rule, and while both interpretations are reasonable and do not allow stealing money, the disagreement would cause the chain to split in half. a serious split of that type happened once in ethereum's history (there have been other much smaller splits where very small portions of the network running old versions of the code forked off). defenders of the single-client approach point to consensus failures as a reason to not have multiple implementations: if there is only one client, that one client will not disagree with itself. their model of how number of clients translates into risk might look something like this: i, of course, disagree with this analysis. the crux of my disagreement is that (i) 2010-style catastrophic bugs matter too, and (ii) you never actually have only one client. the latter point is made most obvious by the bitcoin fork of 2013: a chain split occurred because of a disagreement between two different versions of the bitcoin client, one of which turned out to have an accidental and undocumented limit on the number of objects that could be modified in a single block. hence, one client in theory is often two clients in practice, and five clients in theory might be six or seven clients in practice so we should just take the plunge and go on the right side of the risk curve, and have at least a few different clients. arguments for political decentralization monopoly client developers are in a position with a lot of political power. if a client developer proposes a change, and users disagree, theoretically they could refuse to download the updated version, or create a fork without it, but in practice it's often difficult for users to do that. what if a disagreeable protocol change is bundled with a necessary security update? what if the main team threatens to quit if they don't get their way? or, more tamely, what if the monopoly client team ends up being the only group with the greatest protocol expertise, leaving the rest of the ecosystem in a poor position to judge technical arguments that the client team puts forward, leaving the client team with a lot of room to push their own particular goals and values, which might not match with the broader community? concern about protocol politics, particularly in the context of the 2013-14 bitcoin op_return wars where some participants were openly in favor of discriminating against particular usages of the chain, was a significant contributing factor in ethereum's early adoption of a multi-client philosophy, which was aimed to make it harder for a small group to make those kinds of decisions. concerns specific to the ethereum ecosystem namely, avoiding concentration of power within the ethereum foundation itself provided further support for this direction. in 2018, a decision was made to intentionally have the foundation not make an implementation of the ethereum pos protocol (ie. what is now called a "consensus client"), leaving that task entirely to outside teams. how will zk-evms come in on layer 1 in the future? today, zk-evms are used in rollups. this increases scaling by allowing expensive evm execution to happen only a few times off-chain, with everyone else simply verifying snarks posted on-chain that prove that the evm execution was computed correctly. it also allows some data (particularly signatures) to not be included on-chain, saving on gas costs. this gives us a lot of scalability benefits, and the combination of scalable computation with zk-evms and scalable data with data availability sampling could let us scale very far. however, the ethereum network today also has a different problem, one that no amount of layer 2 scaling can solve by itself: the layer 1 is difficult to verify, to the point where not many users run their own node. instead, most users simply trust third-party providers. light clients such as helios and succinct are taking steps toward solving the problem, but a light client is far from a fully verifying node: a light client merely verifies the signatures of a random subset of validators called the sync committee, and does not verify that the chain actually follows the protocol rules. to bring us to a world where users can actually verify that the chain follows the rules, we would have to do something different. option 1: constrict layer 1, force almost all activity to move to layer 2 we could over time reduce the layer 1 gas-per-block target down from 15 million to 1 million, enough for a block to contain a single snark and a few deposit and withdraw operations but not much else, and thereby force almost all user activity to move to layer 2 protocols. such a design could still support many rollups committing in each block: we could use off-chain aggregation protocols run by customized builders to gather together snarks from multiple layer 2 protocols and combine them into a single snark. in such a world, the only function of layer 1 would be to be a clearinghouse for layer 2 protocols, verifying their proofs and occasionally facilitating large funds transfers between them. this approach could work, but it has several important weaknesses: it's de-facto backwards-incompatible, in the sense that many existing l1-based applications become economically nonviable. user funds up to hundreds or thousands of dollars could get stuck as fees become so high that they exceed the cost of emptying those accounts. this could be addressed by letting users sign messages to opt in to an in-protocol mass migration to an l2 of their choice (see some early implementation ideas here), but this adds complexity to the transition, and making it truly cheap enough would require some kind of snark at layer 1 anyway. i'm generally a fan of breaking backwards compatibility when it comes to things like the selfdestruct opcode, but in this case the tradeoff seems much less favorable. it might still not make verification cheap enough. ideally, the ethereum protocol should be easy to verify not just on laptops but also inside phones, browser extensions, and even inside other chains. syncing the chain for the first time, or after a long time offline, should also be easy. a laptop node could verify 1 million gas in ~20 ms, but even that implies 54 seconds to sync after one day offline (assuming single slot finality increases slot times to 32s), and for phones or browser extensions it would take a few hundred milliseconds per block and might still be a non-negligible battery drain. these numbers are manageable, but they are not ideal. even in an l2-first ecosystem, there are benefits to l1 being at least somewhat affordable. validiums can benefit from a stronger security model if users can withdraw their funds if they notice that new state data is no longer being made available. arbitrage becomes more efficient, especially for smaller tokens, if the minimum size of an economically viable cross-l2 direct transfer is smaller. hence, it seems more reasonable to try to find a way to use zk-snarks to verify the layer 1 itself. option 2: snark-verify the layer 1 a type 1 (fully ethereum-equivalent) zk-evm can be used to verify the evm execution of a (layer 1) ethereum block. we could write more snark code to also verify the consensus side of a block. this would be a challenging engineering problem: today, zk-evms take minutes to hours to verify ethereum blocks, and generating proofs in real time would require one or more of (i) improvements to ethereum itself to remove snark-unfriendly components, (ii) either large efficiency gains with specialized hardware, and (iii) architectural improvements with much more parallelization. however, there is no fundamental technological reason why it cannot be done and so i expect that, even if it takes many years, it will be done. here is where we see the intersection with the multi-client paradigm: if we use zk-evms to verify layer 1, which zk-evm do we use? i see three options: single zk-evm: abandon the multi-client paradigm, and choose a single zk-evm that we use to verify blocks. closed multi zk-evm: agree on and enshrine in consensus a specific set of multiple zk-evms, and have a consensus-layer protocol rule that a block needs proofs from more than half of the zk-evms in that set to be considered valid. open multi zk-evm: different clients have different zk-evm implementations, and each client waits for a proof that is compatible with its own implementation before accepting a block as valid. to me, (3) seems ideal, at least until and unless our technology improves to the point where we can formally prove that all of the zk-evm implementations are equivalent to each other, at which point we can just pick whichever one is most efficient. (1) would sacrifice the benefits of the multi-client paradigm, and (2) would close off the possibility of developing new clients and lead to a more centralized ecosystem. (3) has challenges, but those challenges seem smaller than the challenges of the other two options, at least for now. implementing (3) would not be too hard: one might have a p2p sub-network for each type of proof, and a client that uses one type of proof would listen on the corresponding sub-network and wait until they receive a proof that their verifier recognizes as valid. the two main challenges of (3) are likely the following: the latency challenge: a malicious attacker could publish a block late, along with a proof valid for one client. it would realistically take a long time (even if eg. 15 seconds) to generate proofs valid for other clients. this time would be long enough to potentially create a temporary fork and disrupt the chain for a few slots. data inefficiency: one benefit of zk-snarks is that data that is only relevant to verification (sometimes called "witness data") could be removed from a block. for example, once you've verified a signature, you don't need to keep the signature in a block, you could just store a single bit saying that the signature is valid, along with a single proof in the block confirming that all of the valid signatures exist. however, if we want it to be possible to generate proofs of multiple types for a block, the original signatures would need to actually be published. the latency challenge could be addressed by being careful when designing the single-slot finality protocol. single-slot finality protocols will likely require more than two rounds of consensus per slot, and so one could require the first round to include the block, and only require nodes to verify proofs before signing in the third (or final) round. this ensures that a significant time window is always available between the deadline for publishing a block and the time when it's expected for proofs to be available. the data efficiency issue would have to be addressed by having a separate protocol for aggregating verification-related data. for signatures, we could use bls aggregation, which erc-4337 already supports. another major category of verification-related data is zk-snarks used for privacy. fortunately, these often tend to have their own aggregation protocols. it is also worth mentioning that snark-verifying the layer 1 has an important benefit: the fact that on-chain evm execution no longer needs to be verified by every node makes it possible to greatly increase the amount of evm execution taking place. this could happen either by greatly increasing the layer 1 gas limit, or by introducing enshrined rollups, or both. conclusions making an open multi-client zk-evm ecosystem work well will take a lot of work. but the really good news is that much of this work is happening or will happen anyway: we have multiple strong zk-evm implementations already. these implementations are not yet type 1 (fully ethereum-equivalent), but many of them are actively moving in that direction. the work on light clients such as helios and succinct may eventually turn into a more full snark-verification of the pos consensus side of the ethereum chain. clients will likely start experimenting with zk-evms to prove ethereum block execution on their own, especially once we have stateless clients and there's no technical need to directly re-execute every block to maintain the state. we will probably get a slow and gradual transition from clients verifying ethereum blocks by re-executing them to most clients verifying ethereum blocks by checking snark proofs. the erc-4337 and pbs ecosystems are likely to start working with aggregation technologies like bls and proof aggregation pretty soon, in order to save on gas costs. on bls aggregation, work has already started. with these technologies in place, the future looks very good. ethereum blocks would be smaller than today, anyone could run a fully verifying node on their laptop or even their phone or inside a browser extension, and this would all happen while preserving the benefits of ethereum's multi-client philosophy. in the longer-term future, of course anything could happen. perhaps ai will super-charge formal verification to the point where it can easily prove zk-evm implementations equivalent and identify all the bugs that cause differences between them. such a project may even be something that could be practical to start working on now. if such a formal verification-based approach succeeds, different mechanisms would need to be put in place to ensure continued political decentralization of the protocol; perhaps at that point, the protocol would be considered "complete" and immutability norms would be stronger. but even if that is the longer-term future, the open multi-client zk-evm world seems like a natural stepping stone that is likely to happen anyway. in the nearer term, this is still a long journey. zk-evms are here, but zk-evms becoming truly viable at layer 1 would require them to become type 1, and make proving fast enough that it can happen in real time. with enough parallelization, this is doable, but it will still be a lot of work to get there. consensus changes like raising the gas cost of keccak, sha256 and other hash function precompiles will also be an important part of the picture. that said, the first steps of the transition may happen sooner than we expect: once we switch to verkle trees and stateless clients, clients could start gradually using zk-evms, and a transition to an "open multi-zk-evm" world could start happening all on its own. security advisory [implementation bugs in go and python clients can cause dos – fixed – please update clients] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security advisory [implementation bugs in go and python clients can cause dos – fixed – please update clients] posted by jutta steiner on september 2, 2015 security state transition and consensus issue in geth client causes panic (crash) when processing a (valid) block with a specific combination of transactions, which may cause overall network instability if block is accepted and relayed by unaffected clients thus causing a dos. this may happen in a block that contains transactions which suicide to the block reward address. affected configurations: issue reported for geth.while investigating the issue, related issues were discovered and corrected in pyethereum, hence pyethapp is also affected. c++ clients are unaffected. likelihood: low severity: high complexity: high impact: network instability and dos details: a block containing a specific combination of transactions which include one or more suicide calls, while valid, causes panic crash in go-ethereum client and crash in pyethereum. additional details may be posted when available. effects on expected chain reorganisation depth: none. remedial action taken by ethereum: provision of fixes as below. proposed temporary workaround: switch to unaffected client such as eth (c++). fix:upgrade geth and pyethereum client software. go-ethereum (geth): please note that the current stable version of geth is now 1.1.1; if you are running 1.0 and using a package manager such as apt-get or homebrew the client will be upgraded. if using the ppa: sudo apt-get update then sudo apt-get upgrade if using brew: brew update then brew reinstall ethereum if using a windows binary: download the updated binary. if you are building from source: git pull followed by make geth (please use the master branch commit 8f09242d7f527972acb1a8b2a61c9f55000e955d)   the correct version for this update on ubuntu and osx is geth/v1.1.1-8f09242d pyethereum: users of pyethapp should reinstall > pip install pyethapp --force-reinstall previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the end of the beginning… | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the end of the beginning… posted by gavin wood on may 2, 2015 protocol announcements as part of our tiered release process, we will soon be beginning the final phase prior to our frontier release. so far our ongoing proof-of-concept ix testnet has been running smoothly for around a month with dozens of nodes around the world, well over 200,000 blocks and including clients on several platforms and three different clean-room implementations. however, that's not enough of a test for us. in a phase of the poc-ix testnet called "olympic", the usage of this network will be incentivised through a collection of prizes (in ether, naturally!). we will give advice on what usage should maximise your chances of winning ether, though the precise judgement metrics will be withheld until prize time. suffice it to say, those in the employ of ethereum or its subsidiaries will not be eligible. the sorts of usage we will be awarding include: mining spending gas (in a variety of ways) interesting contracts (variety of opcodes, length of execution time) ... and of course a grand prize for causing a fork! vinay will keep you updated on this but our rough time scale for this is as soon as possible :) ongoing work from my side... as you might notice if you keep an eye on our repositories, development proceeds steadily. alex (alongside felix on the go side) continue to perfect our peer-network infrastructure, with the last few changes having been prototyped recently and a period of review and testing to be carried out prior to our olympic binaries being released. marek's work (alongside fabian on the go side) on the json-rpc testing and web3.js continues to prepare the ground for frontier and lots of lovely ðapps. meanwhile, lefteris has been refactoring our ethash implementation, good for both c++ and go clients and helping make it as efficient and clean as possible. yann and arkadiy have been busy polishing mix and getting the deployment experience as smooth as possible, with the help of avsa, fabian and jutta. christian and liana continue working on solidity, now putting in various optimisations and providing information for gas usage accounting in mix. for my time coding, i've been doing mainly core refactoring and optimisation, getting the internals as smooth and reliable as possible, especially in block importing and transaction processing. having largely completed the our test suite for consensus checking, christoph now continues on his mammoth mission to get full test coverage through the c++ codebase, helped along by dimitri. one of our new starters, vlad, will be helping out with p2p networking and ultimately working his way towards whisper. for the past month or so, edward, our other new starter, has been working on our new developer-resources website for which i have big plans. more on that closer the time of release. that's it for now watch this space for the announcement of olympic binaries and start figuring out how you'll be trying to win some of that ether! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum messaging for the masses (including fathers) via infographic | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum messaging for the masses (including fathers) via infographic posted by anthony di iorio on june 21, 2015 research & development when i started evangelizing bitcoin and blockchain tech back in 2012 my dad was a hard sell. there was the common skepticism and typical counters of “what’s backing it?”, “what can be done with it?” and “what the heck is cryptography?" back then my pitch wasn’t refined and the learning tools had just not quite matured yet...but frankly, i think more of the reason he didn’t grasp it was that he just isn’t technical and doesn’t adopt early tech. i delicately persisted and he eventually came around. before the end of 2012 he had secured a nice stash of coins and as the price rose by many multiples i secured more and more of his ear for future opportunities that might arise. now he didn’t decide to buy bitcoins because he ended up understanding the technology, in fact, i’m still not sure years later if he understands much of the bitcoin nitty-gritty or has wrapped his head around the potential implications that blockchain technology provides. when ethereum came along, he most certainly didn’t understand the concept and why i decided to drop everything in late 2013 and risk sizable capital bootstrapping the project with my then co-founders vitalik, mihai, and charles, and eventually the rest of the founding team, as it exists today. bitcoin is complex to explain, ethereum takes it up a couple notches. it’s not rocket science but sure can seem like it to many people. unheard of terms like as smart contracts, smart property, or dapps can seem daunting to the masses. even terms or words becoming more common such as “cryptocurrency” or “distributed ledger system” can still be unfamiliar to many. let’s face it. this technology is not easy for the majority to grasp and the messaging needs continual refinement and crafting for this to change. rightly so, the majority of focus and current goal of ethereum is to get the tools into the hands of developers and to show them how they can create the products end users will eventually value. less time and resources have been spent explaining ethereum to end consumers, enterprise, and institutions. with weekly announcements of “blockchain integrations” (see nasdaq, ibm, honduran government and countless others) and friendly legislation proposals emerge from countries like canada, there’s obvious increased interest from many sectors and as more integration projects start hitting the news, more and more interest gets sparked from organizations not wanting to get left behind. explaining complex ideas so how do we get ethereum, and in general, complex ideas to the general public? well, whether it's a company trying to communicate new innovations to investors, or an educator teaching a challenging topic in a brief amount of time, the problem is how do you take an abundance of mostly technical information and effectively simplify and present if in an engaging and informative way? one answer is infographics. here are a few benefits of this medium infographics are more eye-catching than printed words since they usually combine images, colors and content that naturally draw the eye and appeal to those with shorter attention spans. infographics are extremely shareable around the web using embed code and are perfect for social network sharing if visually appealing with consistent colors, shapes. messages and logo they provide an effective means of brand awareness bellow is the concept of the first general infographic for ethereum designed for the masses. i’ll allow it to speak for itself and i welcome feedback and questions as to why we crafted the message the way we did or why certain terms or ideas were left out. please feel free to share this and help get the message around the web. this is the first of a few proposed infographics. the next one will focus more on ethereum use cases for various sectors, then one for the more technically inclined. i’m missing father’s day with my dad (and my brother's first fathers day) to be in switzerland on ethereum business. however i did see my dad this week and felt a certain accomplishment when after showing him the infographic he told me: “i finally have a clue what ethereum is and its potential. now what’s all this talk about bitcoin?” happy father’s day to all you fathers out there from ethereum anthony di iorio is a founder of ethereum and currently serves as a consultant and adviser for the project. he is president and founder of decentral and decentral consulting services, offering technology consultancy services that specialize in blockchain and decentralized technology integrations for enterprise, small business and start-ups. anthony is the cryptocurrency adviser at mars discovery district, organizes the toronto ethereum meetup group and dec_tech (decentralized technologies) events and will be lecturing this summer at the university of nicocia's masters program in digital currencies. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum foundation is hiring an executive director | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum foundation is hiring an executive director posted by anthony di iorio on april 8, 2015 organizational stiftung ethereum, the zug, switzerland-based ethereum foundation --ethereum.org, is seeking to immediately fill the executive director position within the foundation. the role involves defining and discharging the day-to-day business of the foundation and leading the organization at the executive level. the successful candidate will have significant experience in corporate finance activities and/or operations.  a technical (software, hardware, math, sciences) background is advantageous but not necessary. the ability to effectively operate in switzerland and globally is essential and the successful candidate will need to either live full time in switzerland or spend a majority of her/his time there. experience in building or running a foundation is a plus. knowledge of the cryptocurrency space is strongly desired but not essential. the desire to creatively and energetically work to define emerging decentralized online economic and social systems is a critical requirement. the director will interact with and support several ethereum divisions that are engaged in software development, communications and community outreach activities. the director will oversee operations in subsidiary divisions, further refining which activities properly belong at the foundation level and which at the operating company level. the director will be supported by staff already in place and with the possibility of hiring additional consulting professionals who have shepherded ethereum through local and global business environments. the director will work closely with ethereum's local and global legal counsel, accounting and human resources firms. coordination of activities with local and global organizations like coincenter, the world wide web consortium (w3c), the cryptocurrency research group (ccrg), self-regulatory bodies and traditional regulatory bodies can be expected to be part of the portfolio. salary will be competitive and commensurate with skills and experience. please send letters of interest and cv in to kelley@ethereum.org previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements vitalik’s research and ecosystem update | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search vitalik’s research and ecosystem update posted by vitalik buterin on october 18, 2015 research & development over the last five days, myself, our executive director ming chan, and several others from the ethereum team and ethereum-based projects and businesses including maker, string/dfinity and consensys have been visiting china and hong kong, where there have been a series of back-to-back conferences and events most notably including the blockchain workshop in hong kong on oct 11-13 and the "first global blockchain summit" in shanghai organized by our friends at wanxiang on oct 15-16, while at the same time continuing to work on our usual tasks of research, development, planning devcon and administrative matters. both conferences proved to be, at least in my own admittedly optimistic estimation, highly positive signs of the growth in the ethereum ecosystem. at the hong kong event, on the first day we participated in semi-closed sessions on issues such as decentralized governance, identity and reputation and regulatory compliance. robin hanson brought up the topic of prediction markets as a form of "low-cost micro-governance": if there are situations where a legal or arbitration process is required in order to ultimately resolve disputes, use a prediction market to provide the result instead, only occasionally escalating to the underlying arbitration mechanism. anyone with private information has the incentive to participate in the prediction market: the parties to the dispute themselves, any third parties involved, and even firms like google applying top-of-the-line machine learning algorithms; in most cases, a result that accurately predicts the result that would be obtained by the underlying court or arbitration system should be attainable very cheaply. the next two days consisted of panel discussions on technical and philosophical topics, people discussing their own projects, and the future of blockchains in areas like reputation and internet of things. ethereum was mentioned many times not through us marketing ourselves, but by many individuals who see ethereum as a worthwhile technology on which to build their applications. the shanghai conference was equally impressive. this was also a two-day event, with the first day featuring back-to-back public speeches and panels and the second day focusing on topic-focused sessions; i personally was involved in moderating the technical discussions on consensus and scalability. this was a fairly large event, perhaps the largest since the global bitcoin summit in beijing which was my first experience with china over one year before and, as has happened over the same period in the west, the audience has transformed to be more mainstream: rather than just employees of bitcoin-specific (or even crypto-specific) projects attending, we saw representatives from major banks, government officials, intel, huawei and others from traditional industry there. people seem excited to see how they could adopt blockchain technology for their own use and there was plenty of interest to go around regarding ethereum specifically. it is indeed true that there are legal restrictions, some fairly clear and some gray, limiting the potential for adoption of bitcoin or cryptocurrencies in china, but there are as yet no issues whatsoever with blockchain technology. the days before and after the conference, we took the time to meet wanxiang as well as the other groups in china that we have regular contact with, particularly both myself and ming meeting for the first time the wonderful team at http://ethfans.org/. we are still determining how we can work with chinese individuals and business and make ethereum as useful as possible in what has become by purchasing power parity the largest economy in the world, but it is clear that the possibilities and the potential are great. research and protocol development we have identified four key areas of research regarding the underlying protocol that we feel represent the key milestones standing in the way between ethereum as it currently exists and crypto-utopia: zk-snark integration: integrating, whether through a new opcode or ideally by leveraging the evm's existing 256-bit modular arithmetic, the ability for ethereum contracts to verify succinct zero-knowledge proofs. given that the verification keys can be generated entirely off-chain, this is somewhat easier than it looks, though once implemented quite a bit of infrastructural work will be needed to make it extremely useful. an initial goal will be to use it to implement ultra-private coin mixing as well as a privacy-preserving reputation ("here's a proof showing that i have a score of over 250 according to your reputation scoring metric and using this set of data published to the blockchain, so you should trust me, but i won't reveal which of the parties that you should trust i am") and two-party financial contracts, and the longer-term goal will be an implementation of hawk on ethereum; at and outside the above-mentioned conferences, we have had some productive chats with some of the developers of these technologies on how to move forward on accomplishing this. casper: ethereum's proof of stake algorithm now under development by vlad zamfir with help from lucius greg meredith, myself and several others. the key components include by-block instead of by-chain consensus and the concept of "economic consensus by bet" as a way of approaching de-facto finality with exponential speed rather than linear speed as is the case with proof of work. the goal is to combine a block time reduction (my personal opinion is that 4 seconds will likely prove to be an good balance between safety and resource overconsumption and centralization risk; vlad is as usual more aggressive), finality and much lower energy consumption (and 10-100x lower price-of-anarchy generally). currently, vlad and greg are working on formally describing and implementing the non-economic part of the algorithm to mathematically prove convergence properties, and the second step will be optimizing the economics. scalability: using a combination of sharding schemes, random sampling, heavy use of merkle proofs and asynchronous calling in order to increase the potential transaction throughput from ~10-20 transactions per second to over 100000 (or, if super-quadratic versions are used, a theoretically unlimited number). the basic concepts behind scaling have been set in stone for over six months, and our research team is highly confident that the general approach is valid; what remains is the details of exactly how to make optimal tradeoffs that preserve as much of ethereum 1.0's functionality as possible in a scalable context. a key design goal is to make sure the ethereum 2.0 remains a superset of ethereum 1.0; for example, synchronous calls should still be possible within a single shard, though cross-shard calls may require asynchrony. i plan on updating my scalability paper to make it more focused, readable and updated with the latest ideas over the next month or so, though in general scalability research is on the back-burner until casper is solidified. evm upgrades: martin becze has taken the lead on exploring webassembly as a candidate for an upgraded ethereum virtual machine. webassembly shares many properties with the evm: the need to run untrusted code, the need for code to be very small, and the need to have multiple compatible implementations, though it lacks gas counting. webassembly can be used to create a just-in-time compiler for the evm, drastically speeding up the ethereum js implementation, and it can also potentially be used as a virtual machine option itself by adding a transformation step that adds a gas subtracting instruction to code before every jump operation. environment opcodes like blockhash, sstore, etc can be abstracted away by turning them into an abi over a generic foreign function interface (eg. sstore(k, v) becomes ffi(0x55 ++ k ++ v) where 0x55; would be the marker representing sstore). aside from this, research questions remain on middleware "on top of" ethereum, including on-chain services, decentralized governance, identity and reputation, random number generation, formal verification for solidity (we have found a talented team in london that is excited about starting to explore the subject) and prediction market implementations, as well as on projects that may live "alongside" ethereum (eg. whisper), though the foundation is leaving it to the community to do much of this extra work as we have deliberately undertaken a strategy of focusing on the core (and doing so in a very lightweight fashion). regarding homestead, the homestead milestone has always been somewhat arbitrary in its definition; unlike frontier, which is marked by a grand and ceremonious launch of the blockchain, and metropolis, which is marked by the release of mist, homestead was always simply meant to be the point where we go from saying "ethereum is unsafe" to "ethereum is somewhat safe". over the last two months, we have released a wallet, alphas for state tree pruning in pyethereum and cpp-ethereum and headers-first syncing in go, upgrades to solidity, very early work in the "light ethereum sub-protocol" (les) and are soon releasing the 1.0 versions of cpp-ethereum and mix. our internal target has been to launch homestead when the frontier network has successfully run without serious problems for 4 weeks (the "serious problem" threshold being downgraded from the frontier definition of "consensus failure" to a more expanded definition which is something like "consensus failure or people losing large amounts of money or semi-serious network glitches"); we plan on announcing a more detailed strategy for homestead soon but suffice it to say that we are already mostly there. devcon is still slated for november 9-13 in london, and we are looking forward to seeing everyone there! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle 「网络国家」之我见 2022 jul 13 see all posts 感谢 付航 ，roy，zik 和其他的greenpillcn社区参与者翻译 7 月 4 日，巴拉吉-斯里尼瓦山（balaji srinivasan）发表了人们 期待已久的新书，描绘他心目中对「网络国家」的愿景：社区纷纷围绕着各自的愿景组织运作起来，起初是各类线上俱乐部，但随着不断树立起越来越大的影响力，最终大到足以寻求政治自治，甚至谋求外交承认。 网络国家可以视为一种尝试，它试图成为自由意志主义思想体系的继承者：巴拉吉本人屡次夸赞《主权个人》一书（参见我的小 读后感），认为它是重要的读物和灵感源泉。但同时，他也与该书的思想在一些主要方面存在分野，集中体现在他在新作中对社会关系中非个人主义和非货币层面的许多论述，比如关于道德准则和社区。网络国家还可以看成是另一种尝试，即：为加密领域勾勒出可能更为宏大的政治叙事。与其在互联网的角落里画地为牢，与广阔的世界脱节，区块链更可以在组织大部分人类社会的新途径中，发挥核心的作用。 这些都可谓是千金之诺。网络国家能够不负所期吗？它是否能切实提供激动人心的裨益？我们先按下网络国家的种种功德不表，把这样一个想法与区块链和加密货币绑定起来，真的说的通吗？另一方面，这样的世界愿景是否遗漏掉了任何至关重要的东西？本文试图展现我对这些问题的思考。 目录 什么是网络国家？ 我们能建设什么样的网络国家 巴拉吉式网络国家的大政治观是什么？ 要接受网络国家就不得不同意巴拉吉的大政治观吗？ 加密货币与网络国家有何关系？ 我喜欢巴拉吉愿景的哪些方面？ 巴拉吉的构想里哪些是我持保留态度的？ 非巴拉吉式的网络国家 有没有中间道路？ 什么是网络国家？ 巴拉吉给出了几个简短的定义，帮助人们理解什么是网络国家。首先，他用一句话定义： 网络国家是目标高度一致的网络社区，有采取集体行动的能力，能够在世界各地众筹领土，最终获得既有国家的外交承认。 上述这句话到目前为止并无争议。创造一个新的网络社区，当成长到足够大的时候，在线下将其实体化，并且最终通过谈判来尝试获得某种地位。在这样一个定义下，无论任何政治意识形态中的人，几乎都能够找寻到自己支持的某些形态的网络国家。但现在，我们来看看他的更长版本的定义： 网络国家就是一个社会网络，它有着道德层面的创新、国民意识和公认的创始人；其集体有行动力，成员个人则文明谦恭；它集成了加密货币，基于共识设立政府，权力由社会智能合约限制；众筹而来的列岛是其真实领土，首都则是虚拟的；通过链上普查来证实其人口、收入和地产规模，使其足以赢得某种程度的外交承认。 此处，这一概念开始得到主观上的强化：我们并不仅仅是在谈论网络社区的一般性概念，不是那些具有集体能动性并且最终尝试在土地上实体化的网络社区；我们所谈论的是在巴拉吉的愿景中网络国家应该是个什么样子。支持广义上的网络国家，但不同意巴拉吉对网络国家应具有何种属性持不同意见是完全有可能的。比如说，如果你还没有成为「加密新信徒」， 就很难理解为什么「集成加密货币」在这个网络国家的概念里如此重要——虽然巴拉吉在稍后的章节里为他的选择进行了辩护。 最后，他将巴拉吉式网络国家的概念扩展成了更长的版本：先是一篇「千字文」（显然，巴拉吉式网络国家采用了 八进制，因为实际字数恰好是 512=8^3）；然后，是一篇 论文，最后，是这本书末尾的 整整一章。 当然，还附了 一张图： image 巴拉吉不惜连篇累牍加以强调的重要观点是，任何成功的新社区都无可避免地需要道德这一成分。他写道： 在这次谈话的 11 分钟处，保罗·约翰逊给出了一个简短的答案。他指出，美洲殖民早期，宗教性殖民地的成功率比盈利性殖民高，因为前者有使命感。略长一点的答案则是，在一个初创社会，你不是去请求人们购买某个产品（经济性的、具有个人主义色彩的营销），而是请求他们加入一个社区（文化层面上的、具有集体主义色彩的宣传）。 宗教社群的承诺悖论（commitment paradox）是关键：与人们的直觉相反，对成员要求最高的宗教社群反而最为持久。 https://vitalik.ca/images/networkstates/communes.png 这就是巴拉吉主义与更加传统的新自由主义—资本主义的截然不同之处——后者的理想型是无害化的、去政治化的、缺乏激情的消费主义「末人」（last man）。与自由论者稻草人式的诡辩不同，巴拉吉并不相信任何东西都「仅仅是一件消费品」。相反，他极力强调社会规范对凝聚力的重要性，以及，让价值观附带一点所谓的宗教性，才能让一个特定网络国家真正有别于外部世界。正如巴拉吉在 这期播客 的 18：20 处所说，如今大多数试图建立私人国家的自由论者就如同「没了犹太教的犹太复国主义」，而这也正是他们失败的关键原因。 这种认识并不是全新的。其实，安东尼奥-加西亚-马丁内斯（ antonio garcia martinez ）对巴拉吉较早前有关主权—个人观点的批评正是着力于此（详见 这档播客约 27:00 处）。巴拉吉曾赞赏那些身处迈阿密的古巴流亡者，称他们坚韧不拔。这些人＂或许不太理性地声称，迈阿密是我们的新家园，是我们最后的庇护所＂。而福山（ fukuyama ）在《历史的终结》一书中指出： 这个城邦，也像现实中的城邦一样，有外敌，需要抵御外部攻击。因此它需要一个护卫者阶层，勇敢且富于公德心，愿意为公共利益牺牲自己的物质欲望。苏格拉底不相信这样的勇敢和公德精神产生于对开明自利的精明算计。相反，它们必然源自激情（thymos），根植于护卫者阶层对其自身和自己的城邦所抱持的正义感、自豪感之中，以及对那些胆敢威胁到这份骄傲的人所可能产生的非理性的愤怒。 巴拉吉在《网络国家》中的观点，我姑且做出这样的阐释：我们的确需要那些不仅由经济利益，也由道德力量绑定在一起的政治团体，虽然如此，我们却不必将自己束缚在现今任何特定的政治团体上，它们已经漏洞百出，而且越来越无法代表民众的价值观。我们能够、而且应该，创造出新的更好的集体，而巴拉吉的 七步走方略 给出了实现方法。 我们能建设什么样的网络国家？ 巴拉吉提出了关于网络国家的一些设想，我将其精简为两个主要方向：沉浸型的生活方式，和倡导科技的监管创新。 关于沉浸型的生活方式，巴拉吉常用为健康而组建的网络国家为例： 接下来，让我们举个例子：假设有一个网络群岛（拥有物理据点），但尚未成为完全的网络国家（未获得外交承认）——我们叫它 keto kosher（无糖社会）。 这得从美国农业部糟糕的食物金字塔说起，这个重谷物的畸形模型，为全球企业高糖化和肥胖症的流行提供了掩护......人们可以组建一个网络社区，在世界各处众筹置产，像是公寓楼、健身房，甚至最后或许是孤街小镇。你没准儿可以实施极端的禁糖措施，完全将加工食品和糖类阻止在边境以外，从而实践某种无糖社会。 你能够想象出这类初创社会的各种不同形态，比如「食肉社区」，或者「史前饮食社区」等。这些都是在同一个大领域内互相竞争的初创社会，在同一个主题下不断迭代。如果成功，这样的一个社会就不会止步于无糖议题。它会开始围绕健身和锻炼设定默认的文化规范。或者，它可能会为社区成员批量采购连续血糖监测仪或订购降血糖药物。 > 严格来说，这些都不需要任何外交承认或者甚至政治自治。然而也许长远来看，这类飞地可能会为其成员展开谋求更低医保费用和医保税的谈判。那么，什么情况下的确需要自治权呢？试想，要设立一个医疗创新自由区会怎么样？ 现在，让我们举一个更难的例子，将会需要一个完整的、获得外交承认的网络国家。那就是医疗主权区——一个没有食品药品监督管理局（fda）的社会。 亨宁格（ henninger ）曾阐述过 fda 所导致的药品延迟上市问题，而塔巴洛克（ tabarrok ）也曾揭示过 fda 对所谓「超药物说明书」处方的干涉。于是，你开创了自己的初创社会。你指出数百万人因为 fda 的政策丧生，像 act－up （译者按：一个呼吁结束艾滋病蔓延的全球组织）那样发放 t 恤，为未来居民播放电影「达拉斯买家俱乐部」，对所有新成员表明你医疗主权事业的正义性...... 对于美国之外的案例，你的初创社会可能会获得，比方说，马耳他 fda 的支持，乘着这股东风设立新的生物医疗体制。在美国国内的话，你将会需要一位州长来宣告成为生物医疗庇护州，就好像一个宣告不会强制执行联邦移民法案的庇护市，一个庇护州将宣告其不会执行 fda 的法令。 > 人们可以为这两类情况做出更多的设想。可以有一个区域，人们可以赤裸身体四处走动。这样的区域一方面保护了你这么做的法定权利，一方面通过创造出人人裸体的环境，让你感到安心。或者，可以有一个区域，所有人只能穿基本款的素色服装，这样就可以打消人们为了比别人好看而大手大脚花钱的念头，从而劝阻这类争夺地位的零和游戏。我们可以设立一个专为加密货币用户提供的国际社区，要求所有店家都接受加密货币，并且干脆需要 nft 才能进入这个区域。或者，我们还可以弄一块飞地，让激进的运输和无人机派送实验合法化，接受更高的个人安全风险，换取参与科技前沿的特权，并有望为全世界设立典范。 所有这些案例的共性就在于，它体现了拥有一个物理区域的价值。至少要有几公顷的土地，让你那个网络国家的独特规则得以实施。当然，你个人可以坚持只在健康的餐厅吃饭，并且去每家餐厅前都认真做攻略。但，如果能划出一袭土地，在这一方地面上，无论去哪儿都能保证你个人的标准得到满足，事情就简单多了。当然，你可以游说当地政府收紧健康和安全法规。但如果这样做，你就得冒着和完全不同偏好的人们发生摩擦的风险，并且存在 将贫困人群排挤出经济系统 的风险。一个网络国家则提供了一条温和的路线。 巴拉吉式网络国家的大政治观是什么？ 这本书有一个引人好奇的特点。读者几乎立即会意识到，它有时像是两本书的结合：有时是一本关于网络国家概念的书，有时则展现出巴拉吉本人宏大的政治观理论。 巴拉吉的大政治观颇为显眼，而且妙趣横生。开篇不久，他就用了一些趣闻来引诱读者......好吧，我直接引用如下： 德国将列宁送回俄罗斯，可能是动摇战时对手的策略的一部分。安东尼-萨顿（antony sutton）的书，则对对某些华尔街银行家如何赞助俄罗斯革命的传闻有所记叙（以及数年后，另一批华尔街银行家又赞助了纳粹）。革命前，托洛茨基在纽约待过一段时间，一些美国人比如约翰-里德（john reed）的宣传性报道帮助了列宁和托洛茨基的革命。事实上，里德给苏联人帮了大忙（扭曲了外界对革命性质的认识），以至于死后被葬在了克里姆林宫墙脚下。是不是感到惊讶：俄国革命并不是完全由俄国人完成，而是有德国人和美国人这些外部力量的深度参与。拥有纽约时报的苏尔兹伯里家族（the ochs-sulzberger family）曾经蓄奴，但在 1619 年的报道中他们却只字未提。鼓吹苏联，掩盖其在乌克兰制造饥荒使之屈服的纽约时报记者沃尔特-杜兰特（walter duranty）却获得了普利策奖，比泰晤士报早了 90 年。 在题为「新闻掺假，何况历史」一章中，你能找到一堆更加耸动的例子。这些例子貌似随性，但其实是刻意为之：其首要目的是将读者从既有的世界观中激震出来，这样他们才能够开始接受巴拉吉自己的理论。 但很快，巴拉吉的举例的确开始指向一些更具体的主题：对美国左翼「觉醒派」的极度厌恶（以纽约时报为代表）；对中共威权主义的强烈不适，与理解其情理之中对美国的惧怕杂糅在一起；他欣赏美国右翼对自由的热爱（以比特币至上主义者为代表），但又不喜欢他们对合作和秩序所抱持的敌意。 接下来，巴拉吉总结了 近代历史上的数次政治重组，最终，我们得到了他的当今政治核心模型：纽约时报（nyt）、中共（ccp）、比特币（btc）构成的三角关系： https://vitalik.ca/images/networkstates/nytccpbtc.png 纽约时报阵营基本上控制着美国，它的无能意味着美国正在走向崩溃。比特币阵营（包括比特币至上主义者和美国泛右翼）有着一些积极的价值观，但对集体行动和秩序的鲜明敌意，让他们无力建设任何东西。中共阵营有能力进行建设，但他们在建设的是一个反乌托邦的监控国家，全世界大部分地方都无意效仿。而且这三方都过于民族主义：他们从自己国家的角度看待事物，对所有其他人只会无视或加以利用。即使理论上都是国际主义者，他们阐释各自价值观的特定方式，仍让人们难以接受，除了世界上属于他们自己的那一小部分人。 在巴拉吉看来，网络国家是 去中心化的中心，它将创造出更好的选择。它结合了比特币阵营对自由的热爱、纽约时报阵营的道德力量，以及中共阵营的组织能力，让我们从这些阵营中尽取精华（外加上任何一方都不具备的国际感召力），摒弃糟粕。 这就是对巴拉吉大政治观的扼要概括。它并不是用一些抽象的理论来为网络国家辩护（比如，一些邓巴数，或者集中激励的观点，认为最理想的政治体应该限制在数万人）。相反，这一观点是将网络国家当成对当前世界具体政治状况的回应。 巴拉吉的历史螺旋论：是的，会有循环周期，但也不断会有进步。当下所处的周期，正需要我们伸出援手，去埋葬僵化的旧秩序， 也要去培育出更美好的新秩序。 https://vitalik.ca/images/networkstates/helical.png 要接受网络国家就不得不同意巴拉吉的大政治观吗？ 许多读者会觉得巴拉吉的大政治观中有很多方面令人心生疑窦。如果你认为保护弱势群体的「觉醒运动」很重要，也许就不会欣赏那种轻慢的态度，认为它只不过是职业精英权力欲的掩饰。如果小弱国家所面临的困境常让你牵挂，比如正在遭受强邻威胁，向外界绝望求助的乌克兰，那么巴拉吉所谓「各国重新武装起来自我防卫，反而可能最好」的 托词 就不能让你信服。 我确实认为，即使不认同巴拉吉的某些逻辑，也仍然可以支持网络国家这一构想，反之亦然。但首先应该解释一下，为什么我认为巴拉吉觉得在问题和解决方案这两者上，他的看法是不可分割的。大致上，巴拉吉对同样的问题热衷已久；在 2013 年所作的题为「终极退出」的演讲中，你就能发现类似表述——即，通过技术的、以退出为主导的措施，来打破美国的制度性僵化。网络国家只是他所提出的解决方案的最新版本。 讨论这个问题之所以重要，有如下理由： 要想表明网络国家是保护自由和资本主义的唯一途径，就必须说明为什么美国做不到。如果美国，或者「自由民主秩序」运转得好好的，就没必要寻找替代品。我们应该只在全球协作和法治上加强建设。但如果美国处于不可逆转的衰退之中，而它的对手正在崛起，事情就相当不同了。那么，网络国家就能够「在不自由的世界里守护自由的价值观」；而臆想「好人当权」的霸权思维则做不到。 巴拉吉的很多目标读者不在美国，而一个网络国家的世界将注定分布于全球——这其中会有许许多多对美国心存疑虑的人。巴拉吉本人是印度裔，拥有大量的印度粉丝。印度内外有很多人并不将美国视为「自由世界秩序的守护者」，相反，往好处说他们觉得美国虚伪，往坏了说则认为美国邪恶。巴拉吉想要表明，你不必是美国的拥趸也可以拥护自由（或至少巴拉吉式的自由）。 很多美国左翼媒体对加密货币和科技行业越来越抱有敌意。在三角关系中，巴拉吉预计纽约时报阵营中的「威权左翼」部分将敌视网络国家，他解释说媒体并非天使，他们的攻击很多出于自利。 但这不是看待宏大图景的唯一方式。假如你确实对社会正义的重要性、纽约时报，和美国有信念呢？假如你重视治理创新，但政治观点较为温和呢？那么，可以用两种方式来看待这件事： 网络国家可以被视为一个协同的战略，或至少是个备份方案。比如说，美国政治中推动平等的任何活动，只能惠及生活在美国的约占全世界 4% 的人口。保护言论自由的宪法第一修正案在美国境外不适用。很多富裕国家的治理已经陷入僵化，我们确实需要尝试一些治理创新的方法。网络国家可以填补这个空白。美国这类国家可以容留很多网络国家，吸引世界各地的人加入。成功的网络国家甚至可以作为政策典范供其他国家效仿。另一方面，设想一下如果 2024 年共和党胜选，并且能够在数十年里拥有多数席位，又或者美国陷入崩解，那该怎么办？你会希望有个备份方案的。 向网络国家撤退是个幌子，甚至是一种威胁。在国内面临重大挑战时，如果所有人的第一反应都是撤退到某块他乡飞地上去，就没有人留下来守护和维持这个国家了。网络国家最终依赖的全球基础设施也将受损。 上述两种视角与很多对巴拉吉式大政治观的异议可以并存。因此，无论支持还是反对巴拉吉式的网络国家，我们终究还是不得不讨论网络国家这个理念。尽管不乏谨慎，而且在网络国家如何运作方面的想法不同，但我个人对它的看法是友善的。 加密货币与网络国家有何关系？ 在两个方面它们双方存在着一致性：精神层面，「比特币已成为科技的旗帜」；实践层面，是指网络国家具体运用区块链和加密代币的方式。总的来说，我同意这两种观点——虽然我认为巴拉吉的书可以将它们更鲜明的表达出来。 精神层面的一致 2022 年，加密货币成了国际主义自由价值观的旗手，而其它堪此重任的社会力量如今难以寻觅。区块链和加密货币本质上就具有全球性。大多数以太坊开发者都在美国以外，住在遥远的欧洲、台湾或澳大利亚之类的地方。nft 已经给 非洲 和其它发展中国家地区的艺术家们带去了独特的机遇。在人类证明（ poh）、kleros 和 nomic labs 等项目上，阿根廷展示了超越自我的实力。 很多地缘政治玩家越来越只顾自身利益，与此同时，区块链社区依旧代表着开放、自由、抗审查和可信中立。这进一步强化了它的全球吸引力：你不必非得钟爱美国霸权，也可以爱上区块链和它背后代表的价值观。这都让区块链成为巴拉吉期望构筑的网络国家愿景的理想精神伴侣。 实践层面的一致 但是，如果离开区块链的实用价值，精神层面的一致就缺乏意义了。巴拉吉给出了大量的区块链应用案例。他青睐的概念之一是将区块链作为「记录账簿」：人们可以在链上给各类事件盖上时间戳，创造出一个全球范围的人类「微观历史」的可信记录。他接着举了其他例子： zcash，ironfish 和 tornado cash 等零知识证明技术，为人们选择公开的内容提供链上证明，而不必披露其它隐私。域名系统，如以太坊域名服务（ens）和索拉纳域名服务（sns）给链上交易附加了身份信息。注册系统则在纯粹的交易层级之上，提供了抽象化公司的链上代表，比如财务报表，甚至是像 dao 这样完全可编程的公司等价物。加密凭证、非同质化代币（nft）、不可转让代币（ntf），以及灵魂绑定代币（soulbunds），允许将非金融数据上链，比如文凭，或者背书。 但所有这些与网络国家又有什么关系？我可以用加密城邦构想内的一些具体例子来加以说明：发行代币；发行 citydao 风格的居民 nft；将区块链与零知识证明加密算法相结合，开展确保隐私保护的投票，还有很多。区块链是加密金融和加密治理的乐高组件：在实现透明的协议规则来治理公共资源、资产和激励方面，它们都是非常有效的工具。 但我们也需要考虑得更深一层。区块链和网络国家有一个共性，他们都试图「创造一个新的根」。一家公司则不是一个根：如果内部发生纠纷，它最终要通过国家的法院系统来解决。另一方面，区块链和网络国家正在试图成为新的底层根系。它并不要求你有那些绝对的「没人能拿我怎么着」的主权式理想，这类的理想大概只有那五个拥有高度自足的国民经济和/或者拥有核武器的国家才能实现。个体的区块链参与者在国家监管面前当然是脆弱的，而网络国家的飞地更有甚之。但区块链至少尝试着成为非国家层面的最终纠纷裁决者（要么通过智能合约的逻辑，要么通过自由分叉）。它是唯一这么做的体系，这让它成为网络国家理想的基础设施。 我喜欢巴拉吉愿景的哪些方面？ 无可避免地，信奉「私产至上」的自由意志主义者遭遇到了一些重大问题，比如在公共物品资助方面表现出的无能为力。有鉴于此，二十一世纪任何成功倡导自由的计划，都不得不成为一个混合体，即，至少包含一个用来解决起码 80% 此类问题的大妥协式方案，而将剩下的问题留给独立个体的积极行动。它可以是某些对抗经济权力和财富集中问题的严格措施（例如，每年对所有物品征收 哈伯格税），可以是 85% 的 乔治主义土地税，也可以是全民基本收入（ubi），还可以强制规模足够大的公司采用内部民主制，或者其他任何提议。不是所有的都能成，但你需要一些激进的东西才能有起码的胜算。 基本上，我习惯了左翼的大妥协方案：某些形式的平等和民主。而另一方面，巴拉吉的大妥协方案感觉上更偏右翼：享有共同价值观的本地社区、 忠诚度、宗教性，构筑物理环境来鼓励个人自律和勤勉工作。这些价值观的实现方式带有很大的自由论和技术进步色彩，它不是围绕着土地、历史、种族和国家来组织的，而是围绕云端和个人选择，虽然如此，却依然是右翼的价值观。这种思考方式对我有些陌生，但我发现它非常迷人，而且重要。典型的「富裕白左」如果忽略了这一点将是他们的损失：这些更加「传统」的价值观事实上非常流行，即使在 一些美国少数族群中 也是如此，而且，在非洲和印度这样的地方受欢迎的程度更甚，这些正是巴拉吉试图建立起根基的地方。 但我这个正在写书评的白左怎么想呢？网络国家真的让我动心了吗？ 我肯定想搬去「keto kosher」——那个沉浸型的、专注健康生活方式的网络国家。当然，我可以专门去找有很多健康方式的城市生活，但一个集中的物理空间让事情变得简单多了。甚至只是周边围绕着有相似目标的人群，这样的激励就足够吸引人了。 但真正有趣的是治理上的创新：运用网络国家实现现行规则下无法实现的组织方式。你可以从三个方面解读其背后的目标： 1、创造新的监管环境来让居民拥有不同于主流的选择。比如，「自由裸体」区域，或者一个在安全和便利之间做出不一样取舍的区域，或者，拥有更多合法精神类药物的区域。 2、创造新的监管机构，使之在相同的关注点上的服务效率高于现有机构。比如，与其通过监管特定的行为来改善环境，可以干脆征收 庇古税（译注：对任何产生负外部性的市场活动征税）。与其用许可证和预审批制来约束诸多事项，可以 强制购买责任险。可以运用 二次方投票 来进行治理，用二次方融资来为当地公共物品提供资助。 3、你所作的任何特定的事，在某些司法官辖区内都可能是允许的。通过增加这样的机会来反抗监管体制中的泛保守主义，比如，机构化了的生物伦理学已经变成了一项臭名昭著的保守事业。在这样的体制下，医学实验出差错死了 20 个人是一场悲剧，但是由于不能及时批准 救命的药品和疫苗，导致 200000 人丧生，却只能是个统计数据。允许人们选择加入能容忍更高风险的网络国家，将是成功的反抗策略。 整体上，在以上所有三个方面我都看到了价值。第一个方面的大规模机构化，可以让世界更加自由的同时，也让人们更容易接受对某些事物更高程度的管制，因为他们知道，总有其它区域可以做一些被禁止的事情。总的来说，我认为第一方面隐含了一个重要的观点：「社会技术型」社区在更好的治理、更好的 公共讨论 方面产生了许多好点子，但如何更好地筛选社会技术却没有得到足够重视。我们并不想单单将现有的社会关系图谱拿来，在其中寻找能让人们取得共识的更好的方法。我们还想对社会关系网本身进行变革，拉近彼此适配的人，创造出更多不同的、能让人们保持各自独特性的生活方式。 第二方面让人振奋，它解决了政治中的一个主要问题。初创企业的早期阶段有些像后期阶段的迷你版，但政治领域不同，其早期阶段是公共论说的游戏，经常选出与实践运作中迥异的东西。如果在网络国家中治理构想能常常付诸实现，我们就可以摆脱巧言善辩的「自由主义空谈者」，转而成为更加平衡的「自由主义实干家」，各种理念的成败将取决于它们在小规模实践中的表现。我们甚至可以将两个方面结合起来：为那些希望每年自动参与一项新治理实验的人们创造一个区域，将其作为一种生活方式。 第三方面当然牵涉到更复杂的道德问题：无论在你看来，陷入瘫痪或正缓慢滑向事实上威权制的全球政府是更大的问题，还是有个什么人发明了邪恶的科技，将我们通通毁灭更危险。我基本上算是第一阵营的；我所担忧的前景，是 西方和中国都变成某种低增长的保守主义；我热衷于看到主权国家之间的协调不善限制了一些事情的执行力度，比如全球版权保护法；我还担忧，未来的监控技术，将导致全球陷入一种高度自我强化的、无法摆脱的可怕的政治均势。但在一些特定的领域（咳咳，怀有敌意的人工智能的风险）我是敬而远之的......但到这里我们算是进入了我反馈的第二部分。 巴拉吉的构想里哪些是我持保留态度的？ 我最担心的有四个方面： 关于创始人——为什么网络国家需要一个公认的创始人，这很关键吗？ 如果网络国家最终只服务于富裕人群怎么办？ 仅仅「退出」不足以稳定全球政治。但如果退出是所有人的第一选择，会发生什么？ 更普遍的全球负外部性怎么办？ 「创始人」这件事 通览全书，巴拉吉坚持认为「创始人」在网络国家中的重要性（或者更确切的说，是初创社会：你建立一个初创社会，直到足够成功获得外交承认变成网络国家）。巴拉吉明确的将初创社会创始人形容为「道德企业家」（moral entrepreneur）： 这些展示很像初创公司的推销文案。但作为初创社会的创始人，你不必像科技公司创业者那样，告诉投资人为何这项创新更好、更快、更便宜。你是一位道德企业家，描述的是潜在的未来，居民有一种更好的生活方式，告诉他们某件事情整个世界都搞错了，而只有你的社区能修正过来。 创始人将道德直觉和从历史中得来的经验教训结晶为具体的哲学，而其道德直觉与该哲学相吻合的人，将会围绕着这个项目凝聚到一起。在早期阶段这很合理——虽然它绝对不是启动初创社会的唯一方式。但发展到后期会发生什么呢？马克-扎克伯格（mark zuckerberg）在脸书的初创阶段作为中心化的创始人或许有其必要，但他作为亿万市值（和数十亿的用户）的公司掌舵人就是另一回事了。或者，巴拉吉的眼中钉，掌管着「纽约时报」的第五代世袭白人奥克斯-苏兹伯格王朝（ochs-sulzberger），又怎么说？ 小东西的中心化很好，但巨大的东西也中心化将非常可怕。鉴于网络效应的现实，再次退出的自由是不充分的。在我看来，如何寻找到创始人控制之外的运转方式很重要，而巴拉吉在这一问题上花的心思太少了。「公认的创始人」被内置在了巴拉吉式网络国家的定义中，但通往广泛参与式治理的路线图却没有。这不应该。 那些不富裕的人群怎么办？ 在过去的一些年里，我们看到了很多国家对于「科技人才」持更加开放的政策。已经有 42 个国家推出了数字游民签证，有 法国科技签证、新加坡的类似项目、台湾的黄金签证、迪拜的 计划 等等等等。这对于有技能的职业人士和富裕人群当然很好。逃离中国科技镇压和防疫封锁（或者是道义上不同意中国政府 其他政策）的百万富翁们，只需要花上几十万美元 买另一本护照，就可以同样躲开 全球对于中国人和其他低收入居民的系统性歧视。但普通人呢？在缅甸面临极端处境的罗兴亚（rohingya）少数民族怎么办，他们中的大多数人并没有一条进入美国或者欧洲的途径，更别说买一本护照了。 在这里，我们看到了网络国家概念中潜在的悲剧。一方面，我真切感受到「退出」是二十一世纪全球人权保护的最可行策略。其他国家在镇压少数族群时你能做什么？你无能为力。你可以制裁他们（但经常是 无效的，甚至给你所关心的人群带来 毁灭性后果）。你可以尝试去侵略（面临同样的指责，甚至更糟）。「退出」是更一个人道的选择，遭受人权迫害的人们可以收拾好行装，去到更友好的栖息地，而且，如果是一群人协调一致行动，还不用牺牲掉赖以维持友谊和经济来源的社区。另一方面，如果你错了，你所批评的政府对人们并没有那么压迫，人们就不会离开，一切都很好，没有饥荒，也没有轰炸。这一切都很美好，除了......当人们想要退出时，没有人在另一边接应他们，整件事情就完蛋了。 答案是什么呢？老实说，我看不到。网络国家的一个好处是他们可以建在贫穷国家，然后吸引海外的富裕人群，这有助于当地经济。但对于想要离开穷国的人来说，这并没有意义。在现有主权国家中以传统的政治行动来推动移民法案松绑，可能是唯一选项。 无处可逃 在俄罗斯 2 月 24 日刚入侵乌克兰的时候，诺亚-史密斯（noah smith）写了一篇重要博文，阐述了这场入侵应该给我们的思想带来道德反省，其中题为「无处可逃」一章尤其令人震撼。引用如下： 虽然「退出」在本地层面有用——如果旧金山运转失衡，你可以搬到奥斯汀或者其他科技城——但在国家层面，这根本行不通，事实上也历来如此——有钱的币圈人搬到比如新加坡这样的国家，或波多黎各这样的地区，但仍然高度依赖于功能强大的政府提供的基础设施和机构。但俄罗斯正在使人们更加清楚地认识到，这种战略注定要失败，因为归根到底，我们无处可逃。跟以往的时代不同，如今大国的长臂管辖，可以触及全世界任何一个角落。 如果美国崩溃，你不能只是搬到新加坡，因为过不了几年，你就得向你的中国新主人屈膝。如果美国崩溃，你也不能只是搬到爱沙尼亚，因为过不了几年（几个月？），你就得向你的俄罗斯新主人屈膝。而这些新主人，将不会有丝毫兴趣保证你的个体自由和财产安全......所以对于自由意志主义者来说，美国不崩溃非常非常重要。 我们有可能面对这样的反驳：确实，如果所有乌克兰人的第一反应都是「退出」，乌克兰早就崩溃了。但如果俄罗斯也更倾向于「退出」，所有俄罗斯人就会在入侵的一周内全都撤离。普京将孤零零地站在卢甘斯克（luhansk）的荒野里，独自面对一百米外的泽连斯基，普京高喊着让泽连斯基投降，泽连斯基回道：「就凭你？你的伙计们在哪儿？」（泽连斯基当然可以在公平的一对一搏斗中胜出）。 但事情也可能走向另一个方向。风险在于，如果润学变成了追求自由的主要方式，社会就会从珍视自由变成润学风行，但中央集权国家将会审查和压制这些冲动，采取对国家无条件忠诚的军国主义态度，并蛮横对待所有其他人。 那些负外部性怎么办？ 如果我们有一百个较少监管的创新实验室，分布在全球各处，可能会导致我们的世界对危险事物更加难以防范。这引申出一个问题：要相信巴拉吉主义，是否必须先相信这个世界的负外部性问题不大？这个观点与「脆弱地球假说」（vulnerable world hypothesis，vwh）相对立，该假说认为，随着科技进步，一个或几个疯子杀死上百万人正在变得越来越容易。为了防范这种极端灾难甚至灭绝场景出现，全球威权式的监控或许成为必需。 专注于个人自卫技术也许是条出路。当然，在一个网络国家的世界里，我们不可能有效禁止功能增益研究（gain-of-function research），但我们也可以运用网络国家开辟一条道路，让这个世界采用优质的 高效空气过滤器、远紫外光、早期检测 基础设施 和快速研发、部署疫苗的通道，不仅能击败新冠，甚至更加凶险的病毒。这个 八万个小时的节目 阐述了一个解决生物武器威胁的乐观场景，但这不是解决所有技术风险的通用方案：至少，并没有自卫的手段可以抵御超级智能的恶意 ai 把我们杀光。 自卫技术当然很好，而且可能是价值受到低估的资助领域。但想仅仅靠这个解决问题并不现实，比如要 禁止屠宰机器人，跨国合作是必需的。所以，我们确定需要的是这样一个世界：即使网络国家比今天的主题社区（intentional community）拥有更多的主权，这个主权也不应该是绝对的。 非巴拉吉式的网络国家 阅读《网络国家》让我想起了十年前读过的另一本书：大卫-德-乌加特（david de ugarte）的《族群：二十一世纪的经济民主》（phyles: economic democracy in the twenty first century）。乌加特讨论了与围绕价值观建立的跨国社区类似的设想，但它有更重的左翼倾向：他假想这些社区将会是民主的，受到 2000 年初的网络社区和十九、二十世纪的合作社及工作场所民主的多重启发。 通过观察乌加特的形成理论，我们能最清晰地看到两者的区别。由于我已经花了大量时间引用巴拉吉，接下来我将给乌加特一段较长引用，以示公平： 博客圈是身份和对话的海洋，处于不断的交叉繁殖和变动之中，而强大的社会消化系统，周期性地提炼出具有自身语境和独特认识的稳定群体。 （这个群体）发展到一定阶段后，形成了一个对话型社区，并在我们所说的数字犹太复国主义（digital zionism）中扮演起了重要角色：他们开始沉淀到现实中，成员之间建立起共同认知，比起想象的共同体（国家、阶级、聚落）中的传统想象，他们更具身份认同感，仿佛这是一个真实的社区（朋友圈、家庭、公会，等等）。 其中一些身份认同强烈且紧密的对话型网络，开始形成他们自己的经济循环系统，且有着清晰的示例——或许不止一个——这让培育社区自身的自主性成为目标，这就是我们所说的「新威尼斯网络」（neo-venetianist networks）。他们诞生于博客圈，是骇客工作伦理的继承者，在理念世界里穿行。这就是我们在本书第一章里提到的经济民主。 与传统的合作主义不同，「新威尼斯网络」并非来自真正的邻近社区，他们的本地连接并不产生认同。比如，在印第安诺基金会（indianos' foundation）中，有位于两个国家和三个自治区的居民，而基金会又是两个相隔百公里开外的公司创立的。 我们可以看到一些非常巴拉吉式的观点：共享的集体身份，但围绕着价值观而不是地理位置，从线上的论坛社区开始，逐步落实成型，进而占据经济生活的一大部分。乌加特甚至和巴拉吉用了一模一样的隐喻——「数字犹太复国主义」。 但我们也能看到关键的区别：没有单一的创始人。初创社会由单个个体的行动来建立——将直觉和思维脉络整合成一套内在一致的正式哲学，而乌加特的族群（phyle）则是开始于博客圈的对话型网络，然后直接转换成一个随时间推移积累越来越深厚的群体——与此同时保持它的民主和横向的性质。整个过程更加有机，完全不是由某个个体的行动引导。 当然，我看到一个直接的挑战，就是这种结构所固有的激励问题。一种对族群（phyles）和网络国家（the network state）可能并不公平的总结是：「网络国家」试图用 2010 年代的区块链技术作为模型来构想如何重新组织人类社会，而「族群」试图采用的模型是 2000 年代的开源软件社区和博客。开源有激励不足的缺陷，而区块链有过度激励和过分集中激励的问题，但这意味着某种中间道路或许是可能的。 有没有中间道路？ 到目前为止，我的判断是网络国家非常出色，但这还远不是一个可行的大妥协方案（big compromise idea），可以填补所有的漏洞，建设起我和我的大部分读者们期待见到的属于二十一世纪的世界。最终，我确实觉得，我们需要引进更多的民主和大规模协同导向的某种大妥协方案，来让网络国家真正取得成功。 以下是我支持的对巴拉吉主义的一些重大调整： 由创始人启动没问题（虽然并非唯一道路），但我们真的需要内置一个「权力移交社区」（exit-to-community）的路线图。 很多创始人希望最终退休，或者开始新事业（基本上一半的加密项目如此），在这种情况下，我们需要能够防止网络国家崩溃，或者滑向平庸。这个过程中的一部分，是宪法将对「权力移交社区」提供保障：当网络国家更成熟、更具规模时，更多社区成员的参与将被自动纳入考量。 prospera（译者注：洪都拉斯的一个特许城市）尝试过类似的事情。斯科特-亚历山大（scott alexander）总结道： 当 prospera 有十万名居民时（现实地说离现在很遥远，即使实验很成功），他们将公投，51% 的大多数可以修改宪章的任何条款，包括将 hpi（译者注：honduras prospera  inc，洪都拉斯 prospera 公司）完全逐出并成为直接民主政体，或者重新加入洪都拉斯，总之可以是任何决定。 但我会青睐更有参与度的治理形式，而不只是居民有一个「是」或「否」的核选项将政府逐出。 这个过程的另一部分，是我在以太坊的成长中认识到的，就是明确地鼓励社成员更广泛地参与社区的理念和哲学发展。以太坊有它的 vitalik，也有它的 polynya：一位网络匿名者，最近他自发地加入进来，开始提供关于扩容技术（rollups and scaling technology）的高质量思考。你的初创社会如何吸引到你的前十位 polynya？ 网络国家的运行应该由非金币驱动（coin-driven）治理来驱动 金币驱动的治理受富人支配，且易受攻击，关于 这一点，我已经 写了 很多 文章 探讨，但还是值得再重复。optimism 的 灵魂绑定代币（soulbound）和每人一份的 市民 nft 是这里的关键。巴拉吉已经承认了非同质化代币的必要性（他 支持金币的锁仓），但我们应该更进一步，更明确地支持超越股东驱动的治理。这也会产生有益的副作用，因为更民主的治理将更好地与外界保持一致。 网络国家通过外界代表参与治理做出友好承诺 在理性主义者和人工智能友好的社区里，有一个迷人但缺乏讨论的想法：可行决策理论。这是一个复杂的概念，但其强大的核心思想是：人工智能通过对其源代码做出可验证的公开承诺，实现比人类更好的相互协调，从而解决人类经常失败的囚徒困境问题。一个人工智能可以重写自己来拥有某个模块，以防止它欺骗具有类似模块的其他人工智能。这样的人工智能将在 囚徒困境中 完全地互相合作。 正如我 几年前指出的，去中心化自治组织（dao）有潜力做到同样的事。他们可以有治理机制来明确对具有类似机制的其他 dao 更加友好。网络国家可以由 dao 来运行，所以这个机制也适用于网络国家。他们甚至可以在治理机制中承诺，将更广泛的公共利益考虑在内（比如，20% 的选票可以给到随机抽选的主办城市或国家的居民），而不用遵循某个特定和复杂的规定，说必须如何考虑这些利益。一个网络国家如此行事的世界，以及主权国家对网络国家采取明确的更友好政策的世界，可能会更加美好。 结语 我期待看到承载着这些愿景的初创社会，期待看到围绕着健康生活的沉浸式的生活方式实验。我也期待看到疯狂的治理实验：其中公共物品由二次方捐赠资助，所有土地区划法规（zoning law）被一个房产税系统取代，基于附近的居民在实时投票系统上表达支持或反对的比例，使房产税率浮动在每年零到五个百分点之间，而投票系统是 由区块链和零知识证明保证 的。我还期待看到更多能承受更高风险的科技实验，前提是取得承受风险的人们的同意。而且我认为基于区块链的通证、身份和声誉系统，以及 dao，将会完美地契合进来。 同时，我担心当前形式的网络国家愿景，只能满足那些足够富有可以自由迁徙，并受到网络国家的热切欢迎的人群，而很多处于社会经济底层的民众将被抛在一边。国际主义是网络国家令人称道的优点：我们甚至有聚焦非洲的 非洲共同体（afropolitan）。国家之间的不平等 占全球不平等的三分之二，国家内部的不平等只占三分之一。但是，对于所有国家中的大量人群，这个愿景仍然与他们无关。所以我们还需要做点别的：为了全球穷人，为了想要守卫自己国家的乌克兰人——他们并不愿意只是被塞进波兰呆上十年，直到波兰也被入侵——以及，为了所有尚未准备好迁往网络国家、或尚未被任何一个网络国家接纳的，所有的人们。 网络国家，还需要做出一些修正，以推动更民主的治理，加强与周边社区的积极关系，或许，还要再加上一些如何帮助大众的方式？这是我愿意追求的愿景。 security advisory [insecurely configured geth can make funds remotely accessible] | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search security advisory [insecurely configured geth can make funds remotely accessible] posted by jutta steiner on august 29, 2015 security insecurely configured ethereum clients with no firewall and unlocked accounts can lead to funds being accessed remotely by attackers. affected configurations: issue reported for geth, though all implementations incl. c++ and python can in principle display this behavior if used insecurely; only for nodes which leave the json-rpc port open to an attacker (this precludes most nodes on internal networks behind nat), bind the interface to a public ip, and simultaneously leave accounts unlocked at startup. likelihood: low severity: high impact: loss of funds related to wallets imported or generated in clients details: it’s come to our attention that some individuals have been bypassing the built-in security that has been placed on the json-rpc interface. the rpc interface allows you to send transactions from any account which has been unlocked prior to sending a transaction and will stay unlocked for the entirety of the the session. by default, rpc is disabled, and by enabling it it is only accessible from the same host on which your ethereum client is running. by opening the rpc to be accessed by anyone on the internet and not including a firewall rules, you open up your wallet to theft by anybody who knows your address in combination with your ip.   effects on expected chain reorganisation depth: none remedial action taken by ethereum: eth rc1 will be fully secure by requiring explicit user-authorisation for any potentially remote transaction. later versions of geth may support this functionality. proposed temporary workaround: only run the default settings for each client and when you do make changes understand how these changes impact your security.   note: this is not a bug, but a misuse of json-rpc.   advisory: never enable json-rpc interface on an internet-accessible machine without a firewall policy in place to block the json-rpc port (default: 8545).   eth: use rc1 or later.   geth: use the safe defaults, and know security implications of the options. --rpcaddr  "127.0.0.1". this is the default value to only allow connections originating on the local computer; remote rpc connections are disabled --unlock. this parameter is used to unlock accounts at startup to aid in automation. by default, all accounts are locked previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum comms announcement | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum comms announcement posted by vitalik buterin on september 18, 2015 organizational the foundation is currently in the phase of restructuring its communications activities. several members of our current communications team in london are soon leaving or reducing their involvement in the foundation in order to pursue for-profit ventures on top of the ethereum ecosystem; we wish them the best of luck. and so, we have both the necessity and a unique opportunity to "reboot" that side of the organization and take another look at how the ethereum foundation interacts with the community. as some initial steps, we are going to do the following: we will make an effort to de-emphasize skype, and emphasize gitter as a means of real-time communication for developers. gitter has the advantages that (i) it's easier to quickly jump in and participate, and (ii) chat rooms are public and visible to everyone by default. currently, the primary active gitter rooms are: https://gitter.im/ethereum/cpp-ethereum https://gitter.im/ethereum/pyethereum https://gitter.im/ethereum/go-ethereum https://gitter.im/ethereum/mist https://gitter.im/ethereum/web3.js we will continue our general direction of increasing permissiveness that we began with our recent changes to branding guidelines, and emphasis on maintaining norms of common sense and civility over strict rules. we are not following in the footsteps of /r/bitcoin's theymos. discussion of ethereumxt (i suppose the closest equivalent we have is expanse, with its 10x higher gas limit) is not banned. we will de-emphasize the "eth dev" brand (with the exception of "devcon") and focus attention on "ethereum" going forward in the medium term, we are in the process of substantially re-examining our strategy internally including meetups, general community outreach, trademark policy, developer documentation, and communicating information on protocol updates (e.g the most pressing short-term updates will be at the networking level to faciliate support of light clients, and larger future changes will be for proof of stake, scalability, etc), as well as interacting with the rapidly growing number of both individual and organizational/corporate ethereum developers outside of the foundation proper. we will continue releasing more concrete info on both this and other topics as it becomes ready, and welcome community input on any of these issues. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum in practice part 1: how to build your own cryptocurrency without touching a line of code | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum in practice part 1: how to build your own cryptocurrency without touching a line of code posted by alex van de sande on december 3, 2015 research & development a lot of things have happened in the past few weeks in the ethereum ecosystem, so many that it might be hard for a casual observer to understand where we are and what's available out there. so i would like to use my first post here to give you an overview of the tools we've built and how you can use them to build interesting things right now. but first a short introduction: i am alex van de sande and i am the lead designer on the ethereum foundation. at devconone i gave a talk entitled "the blockchain versus the average user" where i went into more details on the challenges to bringing the ethereum ecosystem to the aforementioned "average user", one of which is the difficulty of defining what that term even means. when personal computers were introduced they were advertised as being "to the rest of us", but the "rest of us" public they targeted was actually a very narrow audience interested in word processing, spreadsheets and desktop publishing. today, those would be considered among the top tier geeks, just below the kinds of people that still fight over the merits of vim versus emacs. i truly believe that one day your candy-crush-playing grandma/grandpa will use the blockchain, just like today he/she uses databases and https connections without realising it (maybe when they convert blue diamonds won on one game into cows in another game by some other publisher). but right now, what i'm working on is bringing it to that next tier of users, the ones that can download an office suite and figure out how to use a spreadsheet but don't feel comfortable around command lines. so if you consider yourself in that group, then welcome we'll guide you to some basic activities you can do in a few minutes in ethereum: build your own cryptocurrency, building your own democracy and creating a central bank in the blockchain. we skipped the "hello world" tutorial this time, but you'll do fine. download the ethereum wallet download the latest version of the wallet but enough talking, let's get our hands dirty with ethereum! i might be biased but i'd say the best place to start is the ethereum wallet, a multi platform app that will enable you to send ether and ethereum based currencies, create and execute contracts. the first thing you have to do on it is create an ethereum account. all you need for it is a strong password and you'll get it. in order to do anything you'll need to put in some ether: but don't worry you don't need much. most of the contracts here will cost less than a tenth of a us penny. if you are just testing it, we recommend you switch to the testnet: go in the menu develop > network > testnet (morden) and then on develop > start mining. after a few minutes you'll probably have enough ether to test anything, so you can turn it off and save your computer resources. the wallet only allows basic mining on the testnet, but if you want to try your luck on the real net, then you need a more advanced tool. this used to be a cumbersome process but now there are better easier tools: and we have new tools that will make that process much easier. the alethone miner is a straightforward tool with two buttons: press one to start mining in your gpu and press the other to deposit your rewards in a wallet. download it from the turbo suite, a set of power tools created by the c++ team to develop ethereum applications. if you want to create smart contracts on the live network and can't mine you'll need some ethers. you can have a friend sent to you or you can exchange it for bitcoins on a cryptoexchange. if you are a bitcoin fan we suggest you keep on eye on the btcrelay project, a fraud-proof sidechain that will launch soon and allow quick exchanges between ether and bitcoin without a third party. create a token the first contract we are going to create is a token. tokens in the ethereum ecosystem can represent any fungible tradable good: coins, loyalty points, gold certificates, ious, in game items, etc. since all tokens implement some basic features in a standard way, this also means that your token will be instantly compatible with the ethereum wallet and any other client or contract that uses the same standards. go to the contracts page and then click "deploy new contract".   now get the token code from here and paste it into the "solidity source field". if the code compiles without any error, you should see a "pick a contract" drop down on the left. get it and select the "mytoken" contract. on the right column you'll see all the parameters you need to personalize your own token. you can tweak them as you please, but for the purpose of this tutorial we recommend you to pick these parameters: 10,000 as the supply, any name you want, "%" for a symbol and 2 decimal places. your app should be looking like this: scroll to the end of the page and you'll see an estimate of the computation cost of that contract and you can select a fee on how much ether you are willing to pay for it. any excess ether you don't spend will be returned to you so you can leave the default settings if you wish. press "deploy", type your account password and wait a few seconds for your transaction to be picked up. you'll be redirected to the front page where you can see your transaction waiting for confirmations. click the account named "etherbase" (your main account) and after no more than a minute you should see that your account will show that you have 100% of the shares you just created.  to send some to a few friends: select "send", and then choose which currency you want to send (ether or your newly created share), paste your friend's address on the "to" field and press "send". if you send it to a friend, they will not see anything in their wallet yet. this is because the wallet only tracks tokens it knows about, and you have to add these manually. now go to the "contracts" tab and you should see a link for your newly created contract. click on it to go to it's page. since this is a very simple contract page there isn't much to do here, just click "copy address" and paste the contract address on a text editor, you'll need it shortly. to add a token to watch, go to the contracts page and then click "watch token". a popup will appear and you only need to paste the contract address. the token name, symbol and decimal number should be automatically filled but if it's not you can put anything you want (it will only affect how it displays on your wallet). once you do this, you'll automatically be shown any balance you have of that token and you'll be able to send it to anyone else. and now you have your own crypto token! tokens by themselves can be useful as value exchange on local communities, ways to keep track of worked hours or other loyalty programs. but can we make a currency have an intrinsic value by making it useful? tomorrow we'll show how tokens can be used as voting system in order to make collective decisions on the use of funds by creating a democratic autonomous organization. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements final steps | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search final steps posted by stephan tual on july 27, 2015 protocol announcements an update as promised: all systems are now ‘go’ on the technical side (pun intended) and we intend to release frontier this week. thank you to everyone who provided feedback on my previous blog post. what became apparent is that prior to the big day, many of you wanted to know more about what the sequence of events would exactly be, and how to prepare your machine for the release. a transparent and open release frontier users will need to first generate, then load the genesis block into their ethereum client. the genesis block is pretty much a database file: it contains all the transactions from the ether sale, and when a user inputs it into the client, it represents their decision to join the network under its terms: it is the first step to consensus. because the ether pre-sale took place entirely on the bitcoin blockchain, its contents are public, and anyone can generate and verify the genesis block. in the interest of decentralization and transparency, ethereum will not provide the genesis block as a download, but instead has created an open source script that anyone can use to generate the file, a link to which can be found later on in this article. since the script is already available and the release needs to be coordinated, an argument to the script has to be provided in order to 'kick off' frontier in unison. but how can we do this and stay decentralized? the argument needs to be a random parameter that no one, not even us, can predict. as you can imagine, there aren’t too many parameters in the world that match this criteria, but a good one is the hash of a future block on the ethereum testnet. we had to pick a block number, but which one? 1,028,201 turns out to be both prime and palindromic, just the way we like it. so #1028201 is it. sequence of steps to the release: final steps to the release revealed: you are reading this now. block #1028201 is formed on the ethereum tesnet, and is given a hash. the hash is used by users around the world as a unique parameter to the genesis block generation script. what you can do today first, you’ll need the client installed, i’ll use geth as an example, but the same can be achieved with eth (the c++ implementation of ethereum). geth installation instructions for windows, linux and osx can be found on our wiki. once you have installed a client, you need to download the python script that generates the genesis file. it’s called 'mk_genesis_block.py', and can be downloaded here. depending on your platform, you can also download it from the console by installing curl and running; curl -o https://raw.githubusercontent.com/ethereum/genesis_block_generator/master/mk_genesis_block.py this will create the file in the same folder where you invoked the command. you now need to install the pybitcointools created by our very own vitalik buterin. you can obtain this through the python package manager pip, so we’ll install pip first, then the tools right afterwards. the following instructions should work on osx and linux. windows users, good news, pip ships with the standard python installer. curl -o https://bootstrap.pypa.io/get-pip.py sudo python get-pip.py sudo pip install bitcoin or (if you had it installed already), sudo pip install --upgrade bitcoin one last step, if you are using eth, we recently to support the new genesis block parameter, so you'll need to pick up the correct release of the software to be ready for the big day: cd ~/go-ethereum/ git checkout release/1.0.0 git pull make geth those who would like to be ‘as ready as possible’ can follow instructions up to this point, that said, a git pull just before the fateful block is probably recommended to operate the newest version of any software. if you have been running the clients before: back up your keys (maybe some of them are eligible for olympic rewards) they are located in ./ethereum/keystore delete your old chain please (it's located in ./ethereum, delete the following 3 folders only: ./extra, ./state, ./blockchain) you can safely leave your ./ethereum/nodes, ./ethereum/history and ./ethereum/nodekey in place having dags pregenerated in ./ethash will not hurt, but feel free to delete them if you need the space for a complete breakdown as to where the config files are located, please check out this page on our forums. then, it's a matter of waiting for block #1028201, which at the current block resolution time, should be formed approximately thursday evening gmt+0. once 1028201 has formed, its hash will be accessible by querying a node running the testnet using web3.eth.getblock(1028201).hash, however we will also make that value available on this blog as well as all our social media channels. you will then be able to generate the genesis block by running: python mk_genesis_block.py --extradata hash_for_#1028201_goes_here > genesis_block.json by default, the script uses blockr and blockchain.info to fetch the genesis pre-sale results. you can also add the --insight switch if you’d instead prefer to use the private ethereum server to obtain this information. if you are facing issues with the script, please raise an issue on its github. while we will not provide the genesis block as a file, we will still provide the genesis block hash (shortly after we generate it ourselves) in order to insure that third party invalid or malicious files are easily discarded by the community. once you are satisfied with the generation of the genesis block, you can load it into the clients using this command: ./build/bin/geth --genesis genesis_block.json or: ./build/eth/eth --genesis genesis_block.json from there, instructions on creating an account, importing your pre-sale wallet, transacting, etc., can be found on the ‘getting started’ frontier guide at http://guide.ethereum.org/ note that if you've used ethereum before, you should generate new keys using a recent (rc) client, and not reuse testnet keys. a couple more things… we also would like to give you a bit of heads up on the ‘thawing’ phase -the period during which the gas limit per block will be set very low to allow the network to grow slowly before transactions can take place. you should expect network instability at the very beginning of the release, including forks, potential abnormal display of information on our http://stats.ethdev.com page, and various peer to peer connectivity issues. just like during the olympic phase, we expect this instability to settle after a few hours/days. we would also like to remind everyone that while we intend to provide a safe platform in the long term, frontier is a technical release directed at a developer audience, and not a general public release. please keep in mind that early software is often affected by bugs, issues with instability and complex user interfaces. if you would prefer a more user friendly experience, we encourage you to wait for the future homestead or metropolis ethereum releases. please be especially wary of third party websites and software of unknown origin -ethereum will only ever publish software through its github platform at https://github.com/ethereum/. finally, for clarity, it’s important to note that the olympic program ended at block 1m this morning, however, the bug bounty is still on --and will continue until further notice. security vulnerabilities, if found, should continue to be reported to https://bounty.ethdev.com/. -updates 27/07/15: added instructions for users upgrading from previous installations 28/07/15: minor edits, added link to script github 29/07/15: added recommendation to create new keys and not reuse testnet ones previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle the promise and challenges of crypto + ai applications 2024 jan 30 see all posts special thanks to the worldcoin and modulus labs teams, xinyuan sun, martin koeppelmann and illia polosukhin for feedback and discussion. many people over the years have asked me a similar question: what are the intersections between crypto and ai that i consider to be the most fruitful? it's a reasonable question: crypto and ai are the two main deep (software) technology trends of the past decade, and it just feels like there must be some kind of connection between the two. it's easy to come up with synergies at a superficial vibe level: crypto decentralization can balance out ai centralization, ai is opaque and crypto brings transparency, ai needs data and blockchains are good for storing and tracking data. but over the years, when people would ask me to dig a level deeper and talk about specific applications, my response has been a disappointing one: "yeah there's a few things but not that much". in the last three years, with the rise of much more powerful ai in the form of modern llms, and the rise of much more powerful crypto in the form of not just blockchain scaling solutions but also zkps, fhe, (two-party and n-party) mpc, i am starting to see this change. there are indeed some promising applications of ai inside of blockchain ecosystems, or ai together with cryptography, though it is important to be careful about how the ai is applied. a particular challenge is: in cryptography, open source is the only way to make something truly secure, but in ai, a model (or even its training data) being open greatly increases its vulnerability to adversarial machine learning attacks. this post will go through a classification of different ways that crypto + ai could intersect, and the prospects and challenges of each category. a high-level summary of crypto+ai intersections from a ueth blog post. but what does it take to actually realize any of these synergies in a concrete application? the four major categories ai is a very broad concept: you can think of "ai" as being the set of algorithms that you create not by specifying them explicitly, but rather by stirring a big computational soup and putting in some kind of optimization pressure that nudges the soup toward producing algorithms with the properties that you want. this description should definitely not be taken dismissively: it includes the process that created us humans in the first place! but it does mean that ai algorithms have some common properties: their ability to do things that are extremely powerful, together with limits in our ability to know or understand what's going on under the hood. there are many ways to categorize ai; for the purposes of this post, which talks about interactions between ai and blockchains (which have been described as a platform for creating "games"), i will categorize it as follows: ai as a player in a game [highest viability]: ais participating in mechanisms where the ultimate source of the incentives comes from a protocol with human inputs. ai as an interface to the game [high potential, but with risks]: ais helping users to understand the crypto world around them, and to ensure that their behavior (ie. signed messages and transactions) matches their intentions and they do not get tricked or scammed. ai as the rules of the game [tread very carefully]: blockchains, daos and similar mechanisms directly calling into ais. think eg. "ai judges" ai as the objective of the game [longer-term but intriguing]: designing blockchains, daos and similar mechanisms with the goal of constructing and maintaining an ai that could be used for other purposes, using the crypto bits either to better incentivize training or to prevent the ai from leaking private data or being misused. let us go through these one by one. ai as a player in a game this is actually a category that has existed for nearly a decade, at least since on-chain decentralized exchanges (dexes) started to see significant use. any time there is an exchange, there is an opportunity to make money through arbitrage, and bots can do arbitrage much better than humans can. this use case has existed for a long time, even with much simpler ais than what we have today, but ultimately it is a very real ai + crypto intersection. more recently, we have seen mev arbitrage bots often exploiting each other. any time you have a blockchain application that involves auctions or trading, you are going to have arbitrage bots. but ai arbitrage bots are only the first example of a much bigger category, which i expect will soon start to include many other applications. meet aiomen, a demo of a prediction market where ais are players: prediction markets have been a holy grail of epistemics technology for a long time; i was excited about using prediction markets as an input for governance ("futarchy") back in 2014, and played around with them extensively in the last election as well as more recently. but so far prediction markets have not taken off too much in practice, and there is a series of commonly given reasons why: the largest participants are often irrational, people with the right knowledge are not willing to take the time and bet unless a lot of money is involved, markets are often thin, etc. one response to this is to point to ongoing ux improvements in polymarket or other new prediction markets, and hope that they will succeed where previous iterations have failed. after all, the story goes, people are willing to bet tens of billions on sports, so why wouldn't people throw in enough money betting on us elections or lk99 that it starts to make sense for the serious players to start coming in? but this argument must contend with the fact that, well, previous iterations have failed to get to this level of scale (at least compared to their proponents' dreams), and so it seems like you need something new to make prediction markets succeed. and so a different response is to point to one specific feature of prediction market ecosystems that we can expect to see in the 2020s that we did not see in the 2010s: the possibility of ubiquitous participation by ais. ais are willing to work for less than $1 per hour, and have the knowledge of an encyclopedia and if that's not enough, they can even be integrated with real-time web search capability. if you make a market, and put up a liquidity subsidy of $50, humans will not care enough to bid, but thousands of ais will easily swarm all over the question and make the best guess they can. the incentive to do a good job on any one question may be tiny, but the incentive to make an ai that makes good predictions in general may be in the millions. note that potentially, you don't even need the humans to adjudicate most questions: you can use a multi-round dispute system similar to augur or kleros, where ais would also be the ones participating in earlier rounds. humans would only need to respond in those few cases where a series of escalations have taken place and large amounts of money have been committed by both sides. this is a powerful primitive, because once a "prediction market" can be made to work on such a microscopic scale, you can reuse the "prediction market" primitive for many other kinds of questions: is this social media post acceptable under [terms of use]? what will happen to the price of stock x (eg. see numerai) is this account that is currently messaging me actually elon musk? is this work submission on an online task marketplace acceptable? is the dapp at https://examplefinance.network a scam? is 0x1b54....98c3 actually the address of the "casinu inu" erc20 token? you may notice that a lot of these ideas go in the direction of what i called "info defense" in my writings on "d/acc". broadly defined, the question is: how do we help users tell apart true and false information and detect scams, without empowering a centralized authority to decide right and wrong who might then abuse that position? at a micro level, the answer can be "ai". but at a macro level, the question is: who builds the ai? ai is a reflection of the process that created it, and so cannot avoid having biases. hence, there is a need for a higher-level game which adjudicates how well the different ais are doing, where ais can participate as players in the game. this usage of ai, where ais participate in a mechanism where they get ultimately rewarded or penalized (probabilistically) by an on-chain mechanism that gathers inputs from humans (call it decentralized market-based rlhf?), is something that i think is really worth looking into. now is the right time to look into use cases like this more, because blockchain scaling is finally succeeding, making "micro-" anything finally viable on-chain when it was often not before. a related category of applications goes in the direction of highly autonomous agents using blockchains to better cooperate, whether through payments or through using smart contracts to make credible commitments. ai as an interface to the game one idea that i brought up in my writings on is the idea that there is a market opportunity to write user-facing software that would protect users' interests by interpreting and identifying dangers in the online world that the user is navigating. one already-existing example of this is metamask's scam detection feature: another example is the rabby wallet's simulation feature, which shows the user the expected consequences of the transaction that they about to sign. rabby explaining to me the consequences of signing a transaction to trade all of my "bitcoin" (the ticker of an erc20 memecoin whose full name is apparently "harrypotterobamasonic10inu") for eth. edit 2024.02.02: an earlier version of this post referred to this token as a scam trying to impersonate bitcoin. it is not; it is a memecoin. apologies for the confusion. potentially, these kinds of tools could be super-charged with ai. ai could give a much richer human-friendly explanation of what kind of dapp you are participating in, the consequences of more complicated operations that you are signing, whether or not a particular token is genuine (eg. bitcoin is not just a string of characters, it's normally the name of a major cryptocurrency, which is not an erc20 token and which has a price waaaay higher than $0.045, and a modern llm would know that), and so on. there are projects starting to go all the way out in this direction (eg. the langchain wallet, which uses ai as a primary interface). my own opinion is that pure ai interfaces are probably too risky at the moment as it increases the risk of other kinds of errors, but ai complementing a more conventional interface is getting very viable. there is one particular risk worth mentioning. i will get into this more in the section on "ai as rules of the game" below, but the general issue is adversarial machine learning: if a user has access to an ai assistant inside an open-source wallet, the bad guys will have access to that ai assistant too, and so they will have unlimited opportunity to optimize their scams to not trigger that wallet's defenses. all modern ais have bugs somewhere, and it's not too hard for a training process, even one with only limited access to the model, to find them. this is where "ais participating in on-chain micro-markets" works better: each individual ai is vulnerable to the same risks, but you're intentionally creating an open ecosystem of dozens of people constantly iterating and improving them on an ongoing basis. furthermore, each individual ai is closed: the security of the system comes from the openness of the rules of the game, not the internal workings of each player. summary: ai can help users understand what's going on in plain language, it can serve as a real-time tutor, it can protect users from mistakes, but be warned when trying to use it directly against malicious misinformers and scammers. ai as the rules of the game now, we get to the application that a lot of people are excited about, but that i think is the most risky, and where we need to tread the most carefully: what i call ais being part of the rules of the game. this ties into excitement among mainstream political elites about "ai judges" (eg. see this article on the website of the "world government summit"), and there are analogs of these desires in blockchain applications. if a blockchain-based smart contract or a dao needs to make a subjective decision (eg. is a particular work product acceptable in a work-for-hire contract? which is the right interpretation of a natural-language constitution like the optimism law of chains?), could you make an ai simply be part of the contract or dao to help enforce these rules? this is where adversarial machine learning is going to be an extremely tough challenge. the basic two-sentence argument why is as follows: if an ai model that plays a key role in a mechanism is closed, you can't verify its inner workings, and so it's no better than a centralized application. if the ai model is open, then an attacker can download and simulate it locally, and design heavily optimized attacks to trick the model, which they can then replay on the live network. adversarial machine learning example. source: researchgate.net now, frequent readers of this blog (or denizens of the cryptoverse) might be getting ahead of me already, and thinking: but wait! we have fancy zero knowledge proofs and other really cool forms of cryptography. surely we can do some crypto-magic, and hide the inner workings of the model so that attackers can't optimize attacks, but at the same time prove that the model is being executed correctly, and was constructed using a reasonable training process on a reasonable set of underlying data! normally, this is exactly the type of thinking that i advocate both on this blog and in my other writings. but in the case of ai-related computation, there are two major objections: cryptographic overhead: it's much less efficient to do something inside a snark (or mpc or...) than it is to do it "in the clear". given that ai is very computationally-intensive already, is doing ai inside of cryptographic black boxes even computationally viable? black-box adversarial machine learning attacks: there are ways to optimize attacks against ai models even without knowing much about the model's internal workings. and if you hide too much, you risk making it too easy for whoever chooses the training data to corrupt the model with poisoning attacks. both of these are complicated rabbit holes, so let us get into each of them in turn. cryptographic overhead cryptographic gadgets, especially general-purpose ones like zk-snarks and mpc, have a high overhead. an ethereum block takes a few hundred milliseconds for a client to verify directly, but generating a zk-snark to prove the correctness of such a block can take hours. the typical overhead of other cryptographic gadgets, like mpc, can be even worse. ai computation is expensive already: the most powerful llms can output individual words only a little bit faster than human beings can read them, not to mention the often multimillion-dollar computational costs of training the models. the difference in quality between top-tier models and the models that try to economize much more on training cost or parameter count is large. at first glance, this is a very good reason to be suspicious of the whole project of trying to add guarantees to ai by wrapping it in cryptography. fortunately, though, ai is a very specific type of computation, which makes it amenable to all kinds of optimizations that more "unstructured" types of computation like zk-evms cannot benefit from. let us examine the basic structure of an ai model: usually, an ai model mostly consists of a series of matrix multiplications interspersed with per-element non-linear operations such as the relu function (y = max(x, 0)). asymptotically, matrix multiplications take up most of the work: multiplying two n*n matrices takes \(o(n^{2.8})\) time, whereas the number of non-linear operations is much smaller. this is really convenient for cryptography, because many forms of cryptography can do linear operations (which matrix multiplications are, at least if you encrypt the model but not the inputs to it) almost "for free". if you are a cryptographer, you've probably already heard of a similar phenomenon in the context of homomorphic encryption: performing additions on encrypted ciphertexts is really easy, but multiplications are incredibly hard and we did not figure out any way of doing it at all with unlimited depth until 2009. for zk-snarks, the equivalent is protocols like this one from 2013, which show a less than 4x overhead on proving matrix multiplications. unfortunately, the overhead on the non-linear layers still ends up being significant, and the best implementations in practice show overhead of around 200x. but there is hope that this can be greatly decreased through further research; see this presentation from ryan cao for a recent approach based on gkr, and my own simplified explanation of how the main component of gkr works. but for many applications, we don't just want to prove that an ai output was computed correctly, we also want to hide the model. there are naive approaches to this: you can split up the model so that a different set of servers redundantly store each layer, and hope that some of the servers leaking some of the layers doesn't leak too much data. but there are also surprisingly effective forms of specialized multi-party computation. a simplified diagram of one of these approaches, keeping the model private but making the inputs public. if we want to keep the model and the inputs private, we can, though it gets a bit more complicated: see pages 8-9 of the paper. in both cases, the moral of the story is the same: the greatest part of an ai computation is matrix multiplications, for which it is possible to make very efficient zk-snarks or mpcs (or even fhe), and so the total overhead of putting ai inside cryptographic boxes is surprisingly low. generally, it's the non-linear layers that are the greatest bottleneck despite their smaller size; perhaps newer techniques like lookup arguments can help. black-box adversarial machine learning now, let us get to the other big problem: the kinds of attacks that you can do even if the contents of the model are kept private and you only have "api access" to the model. quoting a paper from 2016: many machine learning models are vulnerable to adversarial examples: inputs that are specially crafted to cause a machine learning model to produce an incorrect output. adversarial examples that affect one model often affect another model, even if the two models have different architectures or were trained on different training sets, so long as both models were trained to perform the same task. an attacker may therefore train their own substitute model, craft adversarial examples against the substitute, and transfer them to a victim model, with very little information about the victim. use black-box access to a "target classifier" to train and refine your own locally stored "inferred classifier". then, locally generate optimized attacks against the inferred classfier. it turns out these attacks will often also work against the original target classifier. diagram source. potentially, you can even create attacks knowing just the training data, even if you have very limited or no access to the model that you are trying to attack. as of 2023, these kinds of attacks continue to be a large problem. to effectively curtail these kinds of black-box attacks, we need to do two things: really limit who or what can query the model and how much. black boxes with unrestricted api access are not secure; black boxes with very restricted api access may be. hide the training data, while preserving confidence that the process used to create the training data is not corrupted. the project that has done the most on the former is perhaps worldcoin, of which i analyze an earlier version (among other protocols) at length here. worldcoin uses ai models extensively at protocol level, to (i) convert iris scans into short "iris codes" that are easy to compare for similarity, and (ii) verify that the thing it's scanning is actually a human being. the main defense that worldcoin is relying on is the fact that it's not letting anyone simply call into the ai model: rather, it's using trusted hardware to ensure that the model only accepts inputs digitally signed by the orb's camera. this approach is not guaranteed to work: it turns out that you can make adversarial attacks against biometric ai that come in the form of physical patches or jewelry that you can put on your face: wear an extra thing on your forehead, and evade detection or even impersonate someone else. source. but the hope is that if you combine all the defenses together, hiding the ai model itself, greatly limiting the number of queries, and requiring each query to somehow be authenticated, you can adversarial attacks difficult enough that the system could be secure. in the case of worldcoin, increasing these other defences could also reduce their dependence on trusted hardware, increasing the project's decentralization. and this gets us to the second part: how can we hide the training data? this is where "daos to democratically govern ai" might actually make sense: we can create an on-chain dao that governs the process of who is allowed to submit training data (and what attestations are required on the data itself), who is allowed to make queries, and how many, and use cryptographic techniques like mpc to encrypt the entire pipeline of creating and running the ai from each individual user's training input all the way to the final output of each query. this dao could simultaneously satisfy the highly popular objective of compensating people for submitting data. it is important to re-state that this plan is super-ambitious, and there are a number of ways in which it could prove impractical: cryptographic overhead could still turn out too high for this kind of fully-black-box architecture to be competitive with traditional closed "trust me" approaches. it could turn out that there isn't a good way to make the training data submission process decentralized and protected against poisoning attacks. multi-party computation gadgets could break their safety or privacy guarantees due to participants colluding: after all, this has happened with cross-chain cryptocurrency bridges again and again. one reason why i didn't start this section with more big red warning labels saying "don't do ai judges, that's dystopian", is that our society is highly dependent on unaccountable centralized ai judges already: the algorithms that determine which kinds of posts and political opinions get boosted and deboosted, or even censored, on social media. i do think that expanding this trend further at this stage is quite a bad idea, but i don't think there is a large chance that the blockchain community experimenting with ais more will be the thing that contributes to making it worse. in fact, there are some pretty basic low-risk ways that crypto technology can make even these existing centralized systems better that i am pretty confident in. one simple technique is verified ai with delayed publication: when a social media site makes an ai-based ranking of posts, it could publish a zk-snark proving the hash of the model that generated that ranking. the site could commit to revealing its ai models after eg. a one year delay. once a model is revealed, users could check the hash to verify that the correct model was released, and the community could run tests on the model to verify its fairness. the publication delay would ensure that by the time the model is revealed, it is already outdated. so compared to the centralized world, the question is not if we can do better, but by how much. for the decentralized world, however, it is important to be careful: if someone builds eg. a prediction market or a stablecoin that uses an ai oracle, and it turns out that the oracle is attackable, that's a huge amount of money that could disappear in an instant. ai as the objective of the game if the above techniques for creating a scalable decentralized private ai, whose contents are a black box not known by anyone, can actually work, then this could also be used to create ais with utility going beyond blockchains. the near protocol team is making this a core objective of their ongoing work. there are two reasons to do this: if you can make "trustworthy black-box ais" by running the training and inference process using some combination of blockchains and mpc, then lots of applications where users are worried about the system being biased or cheating them could benefit from it. many people have expressed a desire for democratic governance of systemically-important ais that we will depend on; cryptographic and blockchain-based techniques could be a path toward doing that. from an ai safety perspective, this would be a technique to create a decentralized ai that also has a natural kill switch, and which could limit queries that seek to use the ai for malicious behavior. it is also worth noting that "using crypto incentives to incentivize making better ai" can be done without also going down the full rabbit hole of using cryptography to completely encrypt it: approaches like bittensor fall into this category. conclusions now that both blockchains and ais are becoming more powerful, there is a growing number of use cases in the intersection of the two areas. however, some of these use cases make much more sense and are much more robust than others. in general, use cases where the underlying mechanism continues to be designed roughly as before, but the individual players become ais, allowing the mechanism to effectively operate at a much more micro scale, are the most immediately promising and the easiest to get right. the most challenging to get right are applications that attempt to use blockchains and cryptographic techniques to create a "singleton": a single decentralized trusted ai that some application would rely on for some purpose. these applications have promise, both for functionality and for improving ai safety in a way that avoids the centralization risks associated with more mainstream approaches to that problem. but there are also many ways in which the underlying assumptions could fail; hence, it is worth treading carefully, especially when deploying these applications in high-value and high-risk contexts. i look forward to seeing more attempts at constructive use cases of ai in all of these areas, so we can see which of them are truly viable at scale. ethereum dev update 2015 / week 44 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum dev update 2015 / week 44 posted by taylor gerring on november 2, 2015 research & development with devcon1 a mere week away, the teams are excited and preparing to share all the great tools and technology the ecosystem has to offer. there will be hundreds of developers and dozens of talks including nick szabo, vitalik buterin, and marley gray from microsoft. tickets are limited and we anticipate selling out, so if you haven't already registered, head on over to https://devcon.ethereum.org to secure your spot now! cpp-ethereum turboethereum released 1.0.0 of the c++ tools. work has been done on state trie pruning and upgrading network protocols (pv 62 & 63). several mix bugs have been fixed and the team is working on improving on the build/release process. go-ethereum geth preparing 1.3.0 release and continuing to build new features such as rpc with push support and natural language specification (natspec), allowing users to be be prompted with friendlier confirmation messages. filter search optimizations have been added, speeding up access to historical logs. pyethereum pyethapp is continuing to move forward, with some upgrades to user services and rpc efficiency this week. swarm work towards swarm accounting & incentive mechanisms being finalized along with a whitepaper on results of research. initial integration of swarm with ipfs as data store are beginning in the clients. light client/mobile new network protocols supporting fast sync are rapidly maturing and will be basis for future light clients, helping new users bootstrap into the network within just a few minutes. mist wallet beta 3 available soon with contract deployment and custom token support. both the go and c++ clients are integrated, giving users a choice of consensus implementations. solidity first proof-of-concept of formal verification for solidity contracts now available with why3 tool. additional attention has been given to internal types like tuples. updated documentation online at https://ethereum.github.io/solidity/docs/home/ previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the thawing frontier | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the thawing frontier posted by stephan tual on august 4, 2015 protocol announcements after a smooth launch, please find a few updates of note below. these probably will be of interest only to the more advanced users amongst you if you’re not a miner, there is nothing to action. epoch switch in 2 hours let’s start with a quick heads up to miners, the network will be switching epoch in roughly 2 hours. if you are using the geth/ethminer combo, you should be all set if you have opted to use the --autodag flag on geth and --no-precompute on ethminer. if not, keep an eye on your miners as they generate the dag. report all issues to either the go or c++ issue repositories, depending on which client you’re using. note that some amd cards will require gpu_max_alloc_percent=95 to support the larger dag size. client updates on the horizon. within the next few days, we’ll see the 5k gas limit per block lifted. we’ll be operating with a 3m (3141592, or pi million as it's affectionately called) gas target per block going forward. remember that it’s the protocol that defines the limits by how much the gas limit can increase (or decrease) per block, and that miners can only adjust their parameters within these limits. the max increase (or decrease) is set by the protocol at the parent gas limit /1024. with the default parameter unaffected by miners, it will take a minimum of 6 hours to climb from the current 5k gas per block to a limit allowing transactions (21,000 gas being required for a simple transaction on ethereum). of course, statistically speaking, the more miners update their clients when the update is released, the faster the network will reach the 3m target (the minimum being 28 hours). so if you are mining, keep an eye out for releases of both the c++ and go clients and make sure to update as soon as they hit. default gas price is set at 50 shannon the default gas price is set at 50 shannon (0.05 szabos, or 0.00000005 ether). remember that the gas price is ultimately always defined by the users calling a contract, while the miners can set a parameter on their machines to accept or ignore transactions that have a gas price within a certain range. on the user side of things, we also have a built-in oracle defining the optimal gas price, determined by the previous block gas usage. for example, if the previous block was less than 80% full, the gas price oracle will adjust to a lower gas price recommendation. update on the olympic rewards we are currently compiling the list of olympic winners and will be processing them during the coming weeks. if you are expecting an olympic reward and were not able to do the a->b transaction i covered in one of my previous blog posts, do not panic you can still execute that transaction on the olympic network. some of you reported issues with catching up to the olympic blockchain in this instance, try the c++ client which synchronize in less than 5 hours. note that you may have to mine a few blocks for your transaction to go through, as miners have now quite understandably moved on to frontier. update on namereg once thawing has ended, we'll be looking in adding the name registrar contract. we haven't finalized its behaviour just yet, and its behaviour (similar to an auction house) will probably deserve its own blog post. you can look at its current code published at https://github.com/ethereum/dapp-bin/blob/master/registrar/registrar.sol. announcements will always be made on our blog we thank you for your amazing support and participation on social media, including twitter and of course reddit. note that because so many of you are tracking a few sites for information on ethereum, we decided to release all announcements going forward including change to the protocol or clients on this blog (the one you are reading now). finally, note that if you need answers to technical questions, you'll always be better served by our forums at http://forum.ethereum.org/. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum ðξv: what are we doing? | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum ðξv: what are we doing? posted by gavin wood on december 17, 2014 research & development ok so a minor update about what we are (and are not) doing here at ethereum dev. we are, first and foremost, developing a robust quasi-turing-complete blockchain. this is known as ethereum. aside from having quasi-turing-completeness, it delivers on a number of other important considerations, stemming from the fact we are developing entirely new blockchain technology including: speedy, through a 12 second blocktime; light-client-friendly through the use of merkle roots in headers for compact inclusion/state proofs and dht integration to allow light clients to host & share small parts of the full chain; ðapp-friendly, even for light-clients, through the use of multi-level bloom filters and transaction receipt merkle tries to allow for lightweight log-indexing and proofs; finite-blockchain-friendly we designed the core protocol to facilitate upgrading to this technology, further reducing light-client footprint and helping guarantee mid-term scalability; asic-unfriendly through the (as yet unconfirmed) choice of pow algo and the threat we'll be upgrading to pos in the not-too-distant future. it is robust because: it is unambiguously formally defined, allowing a highly tractable analysis, saturation tests and formal auditing of implementations; it has an extensive, and ultimately complete, set of tests for providing an exceptionally high degree of likelihood a particular implementation is conformant; modern software development practices are observed including a ci system, internal unit tests, strict peer-reviewing, a strict no-warnings policy and automated code analysers; its mesh/p2p backend (aka libp2p) is built on well-tested secure foundations (technology stemming from the kademlia project); official implementations undergo a full industry-standard security audit; a large-scale stress test network will be instituted for profiling and testing against likely adverse conditions and attacks prior to final release. secondly (and at an accordingly lower priority), we are developing materials and tools to make use of this unprecedented technology possible. this includes: developing a single custom-designed co (contract-orientated) language; developing a secure natural language contract specification format and infrastructure; formal documentation for help coding contracts; tutorials for help coding contracts; sponsoring web-based projects in order to get people into development; developing a block chain integrated development environment. thirdly, to facilitate adoption of this technology, gain testers and spur further development we are developing, collaborating over and sponsoring a number of force-multiplying technologies that leverage pre-existing technology including: a graphical client "browser" (leveraging drop-in browser components from the chromium project and qt 5 technology); a set of basic contracts and ðapps, including for registration, reputation, web-of-trust and accounting (leveraging the pre-existing compilers and development tech); a hybrid multi-dht/messaging system, codenamed whisper (leveraging the pre-existing p2p back end & protocols); a simple reverse-hash lookup dht, codenamed swarm (also leveraging the pre-existing p2p back end & protocols), for which there is an ongoing internal implementation, but which could end up merging or being a collaboration with the ipfs project. we are no longer actively targeting multiple languages (lll and mutan are mothballed, serpent is continued as a side project). we are not developing any server technology. and, until there is a working, robust, secure and effective block chain alongside basic development tools, other parts of this overall project have substantially lower priority. following on from the release of the ethereum block chain, expect the other components to get increasingly higher amounts of time dedicated to them. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements an analysis of the first 100000 blocks | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search an analysis of the first 100000 blocks posted by gustav simonsson on august 18, 2015 research & development after a successful launch, frontier passed it's 100000th block earlier this week. here are some interesting stats from these first blocks. the first 50 block times (in seconds): right after genesis, it took 29 seconds for the second block to appear, then 31 and 29 seconds for the next couple of blocks. not too surprising as the miners scrambled with their setups as the final genesis block became known. after these first blocks, we see the block time drop significantly at the onslaught of larger miners, flooring at 1 second that is the limit enforced by the consensus protocol. let's now look at the average block time (every 200th block from genesis to block 10000): we can see the hashing power really attacking the network after the first few hundred blocks, with block times at 1-2 seconds as the difficulty adjustment algorithm is catching up. this algorithm either increases or decreases the difficulty by diff/2048 each block, depending on the block time. after a few thousand blocks, we see the difficulty going up enough for block times to get into the 4-5 second range. after 9k blocks, we're starting to approach the target time of 15 seconds. biggest miners in first 100k blocks (address, blocks, %) : 0xef247e639d49461d25f57e9362cade3120910ce0 851 0.81% 0x790b8a3ce86e707ed0ed32bf89b3269692a23cc1 914 0.87% 0x0037ce3d4b7f8729c8607d8d0248252be68202c0 949 0.90% 0xbb12b5a9b85d4ab8cde6056e9c1b2a4a337d2261 1102 1.04% 0x580992b51e3925e23280efb93d3047c82f17e038 1129 1.07% 0xf2d2aff1320476cb8c6b607199d23175cc595693 1141 1.08% 0x47ff6576639c2e94762ea5443978d7681c0e78dc 1159 1.10% 0x1b7047b4338acf65be94c1a3e8c5c9338ad7d67c 1335 1.26% 0xeb1325c8d9d3ea8d74ac11f4b00f1b2367686319 1446 1.37% 0xbcb2e3693d246e1fc00348754334badeb88b2a11 1537 1.45% 0xa50ec0d39fa913e62f1bae7074e6f36caa71855b 1692 1.60% 0xf8e0ca3ed80bd541b94bedcf259e8cf2141a9523 2437 2.31% 0x9746c7e1ef2bd21ff3997fa467593a89cb852bd0 3586 3.39% 0x88d74a59454f6cf3b51ef6b9136afb6b9d405a88 4292 4.06% 0xbb7b8287f3f0a933474a79eae42cbca977791171 8889 8.41% 0xf927a40c8b7f6e07c5af7fa2155b4864a4112b13 9151 8.66% 0xe6a7a1d47ff21b6321162aea7c6cb457d5476bca 11912 11.28% while one can never know for sure whether a single miner uses multiple coinbase addresses, assuming they use a single one, we have a pretty even distribution of hash power over the first 100k blocks. 0xe6a7a1d47ff21b6321162aea7c6cb457d5476bca with 11% of hashing power is ethpool, the first mining pool for ethereum. next we have two large miners with around 8%. after these three the distribution is quite even, with most miners having less than 1% of total hashing power. however, things develop rapidly in the ethereum world and if we look at the last 15k blocks we see: 0x580992b51e3925e23280efb93d3047c82f17e038 327 2.2% 0xbb7b8287f3f0a933474a79eae42cbca977791171 496 3.3% 0xf927a40c8b7f6e07c5af7fa2155b4864a4112b13 612 4.1% 0x790b8a3ce86e707ed0ed32bf89b3269692a23cc1 674 4.5% 0xe6a7a1d47ff21b6321162aea7c6cb457d5476bca 5775 38.5% here it's clear ethpool is currently having almost 40% of the hashing power, and the 2nd and 3rd biggest miners are down to 4% blocks in a row another interesting stat is to look at continuous sets of blocks from the same miner. this can give some insight into how hashing power and latency is coming into play for larger miners. early after launch, most blocks in a row we saw was 6, for example blocks 1578, 1579, 1580, 1581, 1582 and 1583 mined by 0x9dfc0377058b7b9eb277421769b56df1395705f0. 0xbb7b8287f3f0a933474a79eae42cbca977791171 also mined 6 blocks a few times, for example blocks 656, 657, 658, 659, 660 and 661. this miner also mined 5, 4, 3 and 2 blocks in a row on a number of occasions, not too surprising with 21% of the total hashing power at the time. this happened very early after launch when difficulty was rapidly increasing and many miners had not yet joined the network. as hashing power evened out after block 5000 we did not see more than 4 blocks in a row for some time. however, recently the most blocks in a row is 10, for example blocks 103049, 103050, 103051, 103052, 103053, 103054, 103055, 103056, 103057, 103058 by ethpool. excluding ethpool, no miner has had more than 6 blocks in a row. stay tuned for more statistics from the frontier network as we observe it over the next months! gustav simonsson is a developer in the ethereum security and go teams. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements merkling in ethereum | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search merkling in ethereum posted by vitalik buterin on november 15, 2015 research & development merkle trees are a fundamental part of what makes blockchains tick. although it is definitely theoretically possible to make a blockchain without merkle trees, simply by creating giant block headers that directly contain every transaction, doing so poses large scalability challenges that arguably puts the ability to trustlessly use blockchains out of the reach of all but the most powerful computers in the long term. thanks to merkle trees, it is possible to build ethereum nodes that run on all computers and laptops large and small, smart phones, and even internet of things devices such as those that will be produced by slock.it. so how exactly do these merkle trees work, and what value do they provide, both now and in the future? first, the basics. a merkle tree, in the most general sense, is a way of hashing a large number of "chunks" of data together which relies on splitting the chunks into buckets, where each bucket contains only a few chunks, then taking the hash of each bucket and repeating the same process, continuing to do so until the total number of hashes remaining becomes only one: the root hash. the most common and simple form of merkle tree is the binary mekle tree, where a bucket always consists of two adjacent chunks or hashes; it can be depicted as follows: so what is the benefit of this strange kind of hashing algorithm? why not just concatenate all the chunks together into a single big chunk and use a regular hashing algorithm on that? the answer is that it allows for a neat mechanism known as merkle proofs: a merkle proof consists of a chunk, the root hash of the tree, and the "branch" consisting of all of the hashes going up along the path from the chunk to the root. someone reading the proof can verify that the hashing, at least for that branch, is consistent going all the way up the tree, and therefore that the given chunk actually is at that position in the tree. the application is simple: suppose that there is a large database, and that the entire contents of the database are stored in a merkle tree where the root of the merkle tree is publicly known and trusted (eg. it was digitally signed by enough trusted parties, or there is a lot of proof of work on it). then, a user who wants to do a key-value lookup on the database (eg. "tell me the object in position 85273") can ask for a merkle proof, and upon receiving the proof verify that it is correct, and therefore that the value received actually is at position 85273 in the database with that particular root. it allows a mechanism for authenticating a small amount of data, like a hash, to be extended to also authenticate large databases of potentially unbounded size. merkle proofs in bitcoin the original application of merkle proofs was in bitcoin, as described and created by satoshi nakamoto in 2009. the bitcoin blockchain uses merkle proofs in order to store the transactions in every block: the benefit that this provides is the concept that satoshi described as "simplified payment verification": instead of downloading every transaction and every block, a "light client" can only download the chain of block headers, 80-byte chunks of data for each block that contain only five things: a hash of the previous header a timestamp a mining difficulty value a proof of work nonce a root hash for the merkle tree containing the transactions for that block. if the light client wants to determine the status of a transaction, it can simply ask for a merkle proof showing that a particular transaction is in one of the merkle trees whose root is in a block header for the main chain. this gets us pretty far, but bitcoin-style light clients do have their limitations. one particular limitation is that, while they can prove the inclusion of transactions, they cannot prove anything about the current state (eg. digital asset holdings, name registrations, the status of financial contracts, etc). how many bitcoins do you have right now? a bitcoin light client can use a protocol involving querying multiple nodes and trusting that at least one of them will notify you of any particular transaction spending from your addresses, and this will get you quite far for that use case, but for other more complex applications it isn't nearly enough; the precise nature of the effect of a transaction can depend on the effect of several previous transactions, which themselves depend on previous transactions, and so ultimately you would have to authenticate every single transaction in the entire chain. to get around this, ethereum takes the merkle tree concept one step further. merkle proofs in ethereum every block header in ethereum contains not just one merkle tree, but three trees for three kinds of objects: transactions receipts (essentially, pieces of data showing the effect of each transaction) state this allows for a highly advanced light client protocol that allows light clients to easily make and get verifiable answers to many kinds of queries: has this transaction been included in a particular block? tell me all instances of an event of type x (eg. a crowdfunding contract reaching its goal) emitted by this address in the past 30 days what is the current balance of my account? does this account exist? pretend to run this transaction on this contract. what would the output be? the first is handled by the transaction tree; the third and fourth are handled by the state tree, and the second by the receipt tree. the first four are fairly straightforward to compute; the server simply finds the object, fetches the merkle branch (the list of hashes going up from the object to the tree root) and replies back to the light client with the branch. the fifth is also handled by the state tree, but the way that it is computed is more complex. here, we need to construct what can be called a merkle state transition proof. essentially, it is a proof which make the claim "if you run transaction t on the state with root s, the result will be a state with root s', with log l and output o" ("output" exists as a concept in ethereum because every transaction is a function call; it is not theoretically necessary). to compute the proof, the server locally creates a fake block, sets the state to s, and pretends to be a light client while applying the transaction. that is, if the process of applying the transaction requires the client to determine the balance of an account, the light client makes a balance query. if the light client needs to check a particular item in the storage of a particular contract, the light client makes a query for that, and so on. the server "responds" to all of its own queries correctly, but keeps track of all the data that it sends back. the server then sends the client the combined data from all of these requests as a proof. the client then undertakes the exact same procedure, but using the provided proof as its database; if its result is the same as what the server claims, then the client accepts the proof. patricia trees it was mentioned above that the simplest kind of merkle tree is the binary merkle tree; however, the trees used in ethereum are more complex this is the "merkle patricia tree" that you hear about in our documentation. this article won't go into the detailed specification; that is best done by this article and this one, though i will discuss the basic reasoning. binary merkle trees are very good data structures for authenticating information that is in a "list" format; essentially, a series of chunks one after the other. for transaction trees, they are also good because it does not matter how much time it takes to edit a tree once it's created, as the tree is created once and then forever frozen solid. for the state tree, however, the situation is more complex. the state in ethereum essentially consists of a key-value map, where the keys are addresses and the values are account declarations, listing the balance, nonce, code and storage for each account (where the storage is itself a tree). for example, the morden testnet genesis state looks as follows: { "0000000000000000000000000000000000000001": { "balance": "1" }, "0000000000000000000000000000000000000002": { "balance": "1" }, "0000000000000000000000000000000000000003": { "balance": "1" }, "0000000000000000000000000000000000000004": { "balance": "1" }, "102e61f5d8f9bc71d0ad4a084df4e65e05ce0e1c": { "balance": "1606938044258990275541962092341162602522202993782792835301376" } } unlike transaction history, however, the state needs to be frequently updated: the balance and nonce of accounts is often changed, and what's more, new accounts are frequently inserted, and keys in storage are frequently inserted and deleted. what is thus desired is a data structure where we can quickly calculate the new tree root after an insert, update edit or delete operation, without recomputing the entire tree. there are also two highly desirable secondary properties: the depth of the tree is bounded, even given an attacker that is deliberately crafting transactions to make the tree as deep as possible. otherwise, an attacker could perform a denial of service attack by manipulating the tree to be so deep that each individual update becomes extremely slow. the root of the tree depends only on the data, not on the order in which updates are made. making updates in a different order and even recomputing the tree from scratch should not change the root. the patricia tree, in simple terms, is perhaps the closest that we can come to achieving all of these properties simultaneously. the simplest explanation for how it works is that the key under which a value is stored is encoded into the "path" that you have to take down the tree. each node has 16 children, so the path is determined by hex encoding: for example, the key dog hex encoded is 6 4 6 15 6 7, so you would start with the root, go down the 6th child, then the fourth, and so forth until you reach the end. in practice, there are a few extra optimizations that we can make to make the process much more efficient when the tree is sparse, but that is the basic principle. the two articles mentioned above describe all of the features in much more detail. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the problem of censorship | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the problem of censorship posted by vitalik buterin on june 6, 2015 research & development one of the interesting problems in designing effective blockchain technologies is, how can we ensure that the systems remain censorship-proof? although lots of work has been done in cryptoeconomics in order to ensure that blockchains continue pumping out new blocks, and particularly to prevent blocks from being reverted, substantially less attention has been put on the problem of ensuring that transactions that people want to put into the blockchain will actually get in, even if "the powers that be", at least on that particular blockchain, would prefer otherwise. censorship-resistance in decentralized cryptoeconomic systems is not just a matter of making sure wikileaks donations or silk road 5.0 cannot be shut down; it is in fact a necessary property in order to secure the effective operation of a number of different financial protocols. to take a completely uncontroversial, but high-value, example, consider contracts for difference. suppose that parties a and b both place 100 eth into a contract betting on the gold/usd price, with the condition that if the price after 30 days is $1200, both get 100 eth back, but for every $1 that the price increases a gets 1 eth more and b gets 1 eth less. at the extremes, at $1000 b gets the entire 200 eth, and at $1200 a gets the entire 200 eth. in order for this contract to be a useful hedging tool, one more feature is required: if the price hits $1190 or $1010 at any point during those 30 days, the contract should process immediately, allowing both parties to take out their money and enter another contract to maintain the same exposure (the $10 difference is a safety margin, to give the parties the ability to withdraw and enter a new contract without taking a loss). now, suppose that the price hits $1195, and b has the ability to censor the network. then, b can prevent a from triggering the force-liquidation clause. such a drastic price change likely signals more volatility to come, so perhaps we can expect that when the contract ends there is a 50% chance the price will go back to $1145 and a 50% chance that it will hit $1245. if the price goes back to $1145, then once the contract ends b loses 45 eth. however, if the price hits $1245, then b loses only 100 eth from the price moving $145; hence, b's expected loss is only 72.5 eth and not the 95 eth that it would be if a had been able to trigger the force-liquidation clause. hence, by preventing a from publishing a transaction to the blockchain at that critical time, b has essentially managed to, in common economic and political parlance, privatize the profits and socialize the losses. other examples include auditable computation, where the ability to publish evidence of malfeasance within a particular time-frame is crucial to the mechanism's economic security, decentralized exchanges, where censorship allows users to force others to keep their exchange orders open longer than they intended, and schellingcoin-like protocols, where censors may force a particular answer by censoring all votes that give any other answer. finally, in systems like tendermint, consensus participants can use censorships to prevent other validators from joining the consensus pool, thereby cementing the power of their collusion. hence, all things taken together, anti-censorship is not even about civil liberties; it is about making it harder for consensus participants to engage in large-scale market manipulation conspiracies a cause which seems high on the regulatory agenda. what is the threat model? the first question to ask is, what is the economic model under which we are operating? who are the censors, how much can they do, and how much does it cost them? we will split this up into two cases. in the first case, the censors are not powerful enough to independently block transactions; in the tendermint case, this entails the censors having less than 33% of all validator positions, in which case they can certainly restrict transactions from their own blocks, but those transactions would simply make it into the next block that does not censor them, and that block would still get its requisite 67% signatures from the other nodes. in the second case, the censors are powerful enough; in the bitcoin case, we can think of the top five mining firms and data centers colluding, and in the tendermint case a group of very large stakeholders. this may seem like a silly scenario to worry about after all, many have argued that cryptoeconomic systems rely on a security assumption that such a large group of consensus participants cannot collude, and if they can then we have already lost. however, in those cases, we actually have a secondary defense: such a collusion would destroy the underlying ecosystem and currency, and thus be highly unprofitable to the parties involved. this argument is not perfect; we know that with bribe attacks it's possible for an attacker to set up a collusion where non-participation is a public good, and so all parties will participate even if it is collectively irrational for them, but it nevertheless does set up a powerful defense against one of the more important collusion vectors. with history reversion (ie. 51% attacks), it's clear why carrying out such an attack would destroy the ecosystem: it undermines literally the only guarantee that makes blockchains a single bit more useful than bittorrent. with censorship, however, it is not nearly clear that the same situation applies. one can conceivably imagine a scenario where a large group of stakeholders collude to first undermine specific highly undesirable types of transactions (eg. child porn, to use a popular boogeyman of censors and civil liberties activists complaining about censors alike), and then expand the apparatus over time until eventually it gets into the hands of some enterprising young hotshots that promptly decide they can make a few billion dollars through the cryptoeconomic equivalent of libor manipulation. in the later stages, the censorship may even be done in such a careful and selective way that it can be plausibly denied or even undetected. knowing the results of byzantine fault tolerance theory, there is no way that we can prevent a collusion with more than 33% participation in the consensus process from doing any of these actions absolutely. however, what we can try to do is one of two things: make censorship costly. make it impossible to censor specific things without censoring absolutely everything, or at least without shutting down a very large portion of the features of the protocol entirely. now, let us look at some specific ways in which we can do each one. cost the first, and simplest, way to discourage censorship is a simple one: making it unprofitable, or at least expensive. notably, proof of work actually fails this property: censorship is profitable, since if you censor a block you can (i) take all of its transactions for yourself, and (ii) in the long run take its block reward, as the difficulty adjustment process will reduce difficulty to ensure the block time remains at 10 minutes (or 15 seconds, or whatever) despite the loss of the miner that has been censored away. proof of stake protocols are also vulnerable to (i) by default, but because we can keep track of the total number of validators that are supposed to be participating there are specific strategies that we can take in order to make it less profitable. the simplest is to simply penalize everyone for anyone's non-participation. if 100 out of 100 validators sign a block, everyone gets 100% of the reward. but if only 99 validators sign, then everyone gets 99% of the reward. additionally, if a block is skipped, everyone can be slightly penalized for that as well. this has two sets of consequences. first, censoring blocks produced by other parties will cost the censors. second, the protocol can be designed in such a way that if censorship happens, altruists (ie. default software clients) can refuse to sign the censoring blocks, and thus inflict on the censors an additional expense. of course, some degree of altruism is required for this kind of cost strategy to have any effect if no one was altruistic, then everyone would simply anticipate being censored and not include any undesirable transactions in the first place, but given that assumption it does add substantial costs. timelock consensus as for the second approach, there are two primary strategies that can be undertaken. the first is to use timelock puzzles, a kind of encryption where a piece of data takes a particular amount of time in order to decrypt and which cannot be sped up via parallelization. the typical approach to timelock puzzles is using modular exponentiation; the basic underlying idea is to take a transaction d and generate an encrypted value c with the property: if you know p and q, then computing c from d and d from c are both easy; use the chinese remainder theorem to decompose the problem into: and then use fermat's little theorem to further decompose into: which can be done in a paltry log(n) steps using two rounds of the square-and-multiply algorithm, one for the inner modular exponent and one for the outer modular exponent. one can use the extended euclidean algorithm to compute modular inverses in order to run this calculation backwards. lacking p and q, however, someone would need to literally multiply c by itself n times in order to get the result and, very importantly, the process cannot be parallelized, so it would take just as long for someone with one computer as it would for someone with a thousand. hence, a transaction-sending protocol can be constructed as follows: sender creates transaction t sender encrypts t using p and q to get c, and sends c and pq to a validator alongside a zero-knowledge proof that the values were produced correctly. the validator includes c and pq into the blockchain there is a protocol rule that the validator must submit the correct original transaction t into the blockchain within 24 hours, or else risk losing a large security deposit. honest validators would be willing to participate because they know that they will be able to decrypt the value in time, but they have no idea what they are including into the blockchain until it is too late. under normal circumstances, the sender will also submit t into the blockchain themselves as soon as c is included simply to speed up transaction processing, but if the validators are malicious they will be required to submit it themselves within 24 hours in any case. one can even make the process more extreme: a block is not valid if there remain c values from more than 24 hours ago that have not yet been included. this approach has the advantage that gradual introduction of censorship is impossible outright; it's either all or nothing. however, the "all" is still not that much. the simplest way to get around the mechanism is for validators to simply collude and start requiring senders to send t, p and q alongside c, together with a zero-knowledge proof that all the values are correct. it would be a highly obvious and blatant move, but all in all not a very expensive one. an additional problem of the scheme is that it's highly unnatural, requiring substantial expense of computing power (not nearly as much as proof of work, but nevertheless an hour's worth of computing time on a single core) and slightly non-standard cryptography in order to accomplish. hence, one question is, is there some way in which we can do better? for a simple transaction processing system, the answer is likely no, barring improved versions of timelock that rely on network latency rather than computing power, perhaps in the spirit of andrew miller's nonoutsourceable puzzles. for a turing-complete object model, however, we do have some rather interesting alternatives. a key tool in our arsenal is the halting problem: given a computer program, the only absolutely reliable way to determine what it will do after a number of steps of execution is to actually run it for that long (note: the original formulation asks only whether the program will halt, but the inherent impossibility can be generalized to very many types of output and intermediate behavior). in the context of ethereum, this opens up a particular denial-of-service attack vector: if a censor wishes to block transactions that have an undesirable effect (eg. sending messages to or from a particular address), then that effect could appear after running for millions of computational steps, and so the censor would need to process every transaction and discard the ones that they want censored. normally, this is not a problem for ethereum: as long as a transaction's signature is correct, the transaction is well-formatted and there is enough ether to pay for it, the transaction is guaranteed to be valid and includable into the blockchain, and the including miner is guaranteed to get a reward proprtional to the amount of computation that the transaction is allowed to take up. here, however, the censor is introducing an additional artificial validity condition, and one that cannot be verified nearly so "safely". however, we cannot immediately assume that this denial-of-service vulnerability will be fatal: it only takes perhaps a tenth of a second to verify a maximally sized transaction, and one certainly can overcome attacks of that size. hence, we need to go a step further, and introduce an upcoming ethereum 1.1 feature: events. events are a feature that allows a contract to create a kind of delayed message that is only played at some prespecified block in the future. once an event is made, any block at the height at which the event is supposed to mature must play the event in order to be valid. hence, transaction senders can be clever, and create a hundred transactions that create a hundred events, only all of which together create an event that accomplishes some particular action that is not desired by censors. even now, censors trying to produce their blocks can still try to simulate a series of empty blocks following the block they are producing, to see if the sequence of events that they are generating will lead to any undesirable consequence. however, transaction senders can make life much harder for censors still: they can create sets of transactions that create events that don't by themselves do anything, but do lead to the sender's desired consequence in combination with some other transaction that happens regularly (eg. bloomberg publishing some data feed into their blockchain contract). relying on block timestamps or other unpredictable block data is another possibility. note that this also makes it much harder to enact another defense against these anti-censorship strategies: requiring transaction senders themselves to produce a zero-knowledge proof that their transactions bear no undesirable intent. to expand the functionality of this scheme, we can also add another protocol feature: create a specialized address where messages sent to that address are played as transactions. the messages would contain the transaction data in some form (eg. each message specifies one byte), after a few hundred blocks trigger events to combine the data together, and the data would then have to be immediately played as a regular transaction; once the initial transactions are in, there is no way around it. this would basically ensure that everything that can be done by sending transactions (the primary input of the system) can be done through this kind of covert latent message scheme. hence, we can see how blocking such circumventions will very likely be pretty much impossible to do completely and absolutely; rather, it will be likely a constant two-sided war of heuristics versus heuristics where neither side would have a permanent upper hand. we may see the development of centralized firms whose sole purpose is to accept any transaction and find some way to "sneak it in" to the blockchain in exchange for a fee, and these firms would consistently update their algorithms in response to the updated algorithms of the parties that are trying to work against their previous algorithms to block the attempt. perhaps, this is the best that we can do. anti-censorship and finality it is important to note that the above by itself does not prove that censorship is extremely expensive all on its own. rather, it shows that, if developers take care to add certain features into the blockchain protocol, censorship can be made as hard as reversion. this still leaves the question of how difficult reversion is in the first place. a lot of earlier consensus protocols, including proof of work and naive versions of proof of stake, do not make small-depth reversion very difficult; hence, if it takes a hundred blocks to realize that an undesirable transaction has successfully entered the system, then it would be a major inconvenience but the validators would be able to discard the old blockchain and create a new one, with all of the transactions from the old chain included in order in order to avoid inconveniencing anyone else (although anyone that was using the blockchain as a source of randomness would unfortunately be out of their luck). newer protocols like tendermint, however, use security deposits to make reverting even one block almost impossible, and so do not run into this problem; if you can get the delayed events into the blockchain at all, you've already won. this, incidentally, is an important case study of the importance of "bribe attacks" as a theoretical concern in cryptoeconomics: even though literal bribes may in many cases be unrealistic, external incentive adjustments can come from any source. if one can prove that blockchains are extremely expensive to revert, then one can be assured that they will be extremely expensive to revert for any purpose, including attacker bribes and external desires to revert transactions for some particular purpose. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements a few last minute notes… | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search a few last minute notes… posted by stephan tual on july 30, 2015 protocol announcements genesis is roughly 4 hours away, here are a few last minute pointers for those preparing: all clients are functional on all platforms and build to completion: http://ethereum-buildboard.meteor.com/ please note that it will take a little while (10-1000 blocks) before the network stabilises. this is a marathon, not a sprint. for this reason we will not make the genesis block hash known until a solid consensus has been formed. we will not announce the --extradata immediately either if we did, we might as well launch centrally. we'll let the community come to a consensus on what it is (as there could be multiple blocks #1028201 on the testnet, while it undergoes a re-org, or multiple re-orgs). do not reuse keys from olympic, especially if you are expecting olympic rewards, as these should be considered completely insecure due to the development state of the various codebases that generated them. they may also be vulnerable to replay attacks. if you are expecting an olympic reward for a key a, please create new key b with pyethereum, cpp-ethereum-0.9.36+ or go-ethereum-1.0 and place a transaction sent by a to b on the testnet; rewards will be given to the first such transaction following block #1026900. don't use b on olympic. if you have 'won' something, it will get sent to b on frontier. difficulty will start at 2^34, this number was decided on based on what we observed on the testnet see you at genesis! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements introducing ethereum script 2.0 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search introducing ethereum script 2.0 posted by vitalik buterin on february 3, 2014 research & development this post will provide the groundwork for a major rework of the ethereum scripting language, which will substantially modify the way es works although still keeping many of the core components working in the exact same way. the rework is necessary as a result of multiple concerns which have been raised about the way the language is currently designed, primarily in the areas of simplicity, optimization, efficiency and future-compatibility, although it does also have some side-benefits such as improved function support. this is not the last iteration of es2; there will likely be many incremental structural improvements that can be made to the spec, but it does serve as a strong starting point. as an important clarification, this rework will have little effect on the ethereum cll, the stripped-down-python-like language in which you can write namecoin in five lines of code. the cll will still stay the same as it is now. we will need to make updates to the compiler (an alpha version of which is now available in python at http://github.com/ethereum/compiler or as a friendly web interface at http://162.218.208.138:3000) in order to make sure the cll continues to compile to new versions of es, but you as an ethereum contract developer working in e-cll should not need to see any changes at all. problems with es1 over the last month of working with es1, several problems with the language’s design have become apparent. in no particular order, they are as follows: too many opcodes – looking at the specification as it appears today, es1 now has exactly 50 opcodes – less than the 80 opcodes found in bitcoin script, but still far more than the theoretically minimal 4-7 opcodes needed to have a functional turing-complete scripting language. some of those opcodes are necessary because we want the scripting language to have access to a lot of data – for example, the transaction value, the transaction source, the transaction data, the previous block hash, etc; like it or not, there needs to be a certain degree of complexity in the language definition to provide all of these hooks. other opcodes, however, are excessive, and complex; as an example, consider the current definition of sha256 or ecverify. with the way the language is designed right now, that is necessary for efficiency; otherwise, one would have to write sha256 in ethereum script by hand, which might take many thousands of basefees. but ideally, there should be some way of eliminating much of the bloat. not future-compatible – the existence of the special crypto opcodes does make es1 much more efficient for certain specialized applications; thanks to them, computing sha3 takes only 40x basefee instead of the many thousands of basefees that it would take if sha3 was implemented in es directly; same with sha256, ripemd160 and secp256k1 elliptic curve operations. however, it is absolutely not future-compatible. even though these existing crypto operations will only take 40x basefee, sha4 will take several thousand basefees, as will ed25519 signatures, the quantum-proofntru, scip and zerocoin math, and any other constructs that will appear over the coming years. there should be some natural mechanism for folding such innovations in over time. not deduplication-friendly – the ethereum blockchain is likely to become extremely bloated over time, especially with every contract writing its own code even when the bulk of the code will likely be thousands of people trying to do the exact same thing. ideally, all instances where code is written twice should pass through some process of deduplication, where the code is only stored once and only a pointer to the code is stored twice. in theory, ethereum’s patricia trees do this already. in practice, however, code needs to be in exactly the same place in order for this to happen, and the existence of jumps means that it is often difficult to abitrarily copy/paste code without making appropriate modifications. furthermore, there is no incentivization mechanism to convince people to reuse existing code. not optimization-friendly – this is a very similar criterion to future-compatibility and deduplication-friendliness in some ways. however, here optimization refers to a more automatic process of detecting bits of code that are reused many times, and replacing them with memoized or compiled machine code versions. beginnings of a solution: deduplication the first issue that we can handle is that of deduplication. as described above, ethereum patricia trees provide deduplication already, but the problem is that achieving the full benefits of the deduplication requires the code to be formatted in a very special way. for example, if the code in contract a from index 0 to index 15 is the same as the code in contract b from index 48 to index 63, then deduplication happens. however, if the code in contract b is offset at all modulo 16 (eg. from index 49 to index 64), then no deduplication takes place at all. in order to remedy this, there is one relatively simple solution: move from a dumb hexary patricia tree to a more semantically oriented data structure. that is, the tree represented in the database should mirror the abstract syntax tree of the code. to understand what i am saying here, consider some existing es1 code: txvalue push 25 push 10 push 18 exp mul lt not push 14 jmpi stop push 0 txdata sload not push 0 txdata push 1000 lt not mul not not push 32 jmpi stop push 1 txdata push 0 txdata sstore in the patricia tree, it looks like this: ( (txvalue push 25 push 10 push 18 exp mul lt not push 14 jmpi stop push) (0 txdata sload not push 0 txdata push 1000 lt not mul not not push 32) (jmpi stop push 1 txdata push 0 txdata sstore) ) and here is what the code looks like structurally. this is easiest to show by simply giving the e-cll it was compiled from: if tx.value < 25 * 10^18: stop if contract.storage[tx.data[0]] or tx.data[0] < 1000: stop contract.storage[tx.data[0]] = tx.data[1] no relation at all. thus, if another contract wanted to use some semantic sub-component of this code, it would almost certainly have to re-implement the whole thing. however, if the tree structure looked somewhat more like this: ( ( if (txvalue push 25 push 10 push 18 exp mul lt not) (stop) ) ( if (push 0 txdata sload not push 0 txdata push 1000 lt not mul not) (stop) ) ( push 1 txdata push 0 txdata sstore ) ) then if someone wanted to reuse some particular piece of code they easily could. note that this is just an illustrative example; in this particular case it probably does not make sense to deduplicate since pointers need to be at least 20 bytes long to be cryptographically secure, but in the case of larger scripts where an inner clause might contain a few thousand opcodes it makes perfect sense. immutability and purely functional code another modification is that code should be immutable, and thus separate from data; if multiple contracts rely on the same code, the contract that originally controls that code should not have the ability to sneak in changes later on. the pointer to which code a running contract should start with, however, should be mutable. a third common optimization-friendly technique is the make a programming language purely functional, so functions cannot have any side effects outside of themselves with the exception of return values. for example, the following is a pure function: def factorial(n): prod = 1 for i in range(1,n+1): prod *= i return prod however, this is not: x = 0 def next_integer(): x += 1 return x and this most certainly is not: import os def happy_fluffy_function(): bal = float(os.popen('bitcoind getbalance').read()) os.popen('bitcoind sendtoaddress 1jwssubhmg6iptrjtyqhuyyh7bzg3lfy1t %.8f' % (bal 0.0001)) os.popen('rm -rf ~') ethereum cannot be purely functional, since ethereum contracts do necessarily have state – a contract can modify its long-term storage and it can send transactions. however, ethereum script is a unique situation because ethereum is not just a scripting environment – it is an incentivized scripting environment. thus, we can allow applications like modifying storage and sending transactions, but discourage them with fees, and thus ensure that most script components are purely functional simply to cut costs, even while allowing non-purity in those situations where it makes sense. what is interesting is that these two changes work together. the immutability of code also makes it easier to construct a restricted subset of the scripting language which is functional, and then such functional code could be deduplicated and optimized at will. ethereum script 2.0 so, what’s going to change? first of all, the basic stack-machine concept is going to roughly stay the same. the main data structure of the system will continue to be the stack, and most of your beloved opcodes will not change significantly. the only differences in the stack machine are the following: crypto opcodes are removed. instead, we will have to have someone write sha256, ripemd160, sha3 and ecc in es as a formality, and we can have our interpreters include an optimization replacing it with good old-fashioned machine-code hashes and sigs right from the start. memory is removed. instead, we are bringing back dupn (grabs the next value in the code, say n, and pushes a copy of the item n items down the stack to the top of the stack) and swapn (swaps the top item and the nth item). jmp and jmpi are removed. run, if, while and setroot are added (see below for further definition) another change is in how transactions are serialized. now, transactions appear as follows: send: [ 0, nonce, to, value, [ data0 ... datan ], v, r, s ] mkcode: [ 1, nonce, [ data0 ... datan ], v, r, s ] mkcontract: [ 2, nonce, coderoot, v, r, s ] the address of a contract is defined by the last 20 bytes of the hash of the transaction that produced it, as before. additionally, the nonce no longer needs to be equal to the nonce stored in the account balance representation; it only needs to be equal to or greater than that value. now, suppose that you wanted to make a simple contract that just keeps track of how much ether it received from various addresses. in e-cll that’s: contract.storage[tx.sender] = tx.value in es2, instantiating this contract now takes two transactions: [ 1, 0, [ txvalue txsender sstore ], v, r, s] [ 2, 1, 761fd7f977e42780e893ea44484c4b64492d8383, v, r, s ] what happens here is that the first transaction instantiates a code node in the patricia tree. the hash sha3(rlp.encode([ txvalue txsender sstore ]))[12:] is 761fd7f977e42780e893ea44484c4b64492d8383, so that is the “address” where the code node is stored. the second transaction basically says to initialize a contract whose code is located at that code node. thus, when a transaction gets sent to the contract, that is the code that will run. now, we come to the interesting part: the definitions of if and run. the explanation is simple: if loads the next two values in the code, then pops the top item from the stack. if the top item is nonzero, then it runs the code item at the first code value. otherwise, it runs the code item at the second code value. while is similar, but instead loads only one code value and keeps running the code while the top item on the stack is nonzero. finally, run just takes one code value and runs the code without asking for anything. and that’s all you need to know. here is one way to do a namecoin contract in new ethereum script: a: [ txvalue push 25 push 10 push 18 exp mul lt ] b: [ push 0 txdata sload not push 0 txdata push 100 lt not mul not ] z: [ stop ] y: [ ] c: [ push 1 txdata push 0 txdata sstore ] m: [ run a if z y run b if z y run c ] the contract would then have its root be m. but wait, you might say, this makes the interpreter recursive. as it turns out, however, it does not – you can simulate the recursion using a data structure called a “continuation stack”. here’s what the full stack trace of that code might look like, assuming the transaction is [ x, y ] sending v where x > 100, v > 10^18 * 25and contract.storage[x] is not set: { stack: [], cstack: [[m, 0]], op: run } { stack: [], cstack: [[m, 2], [a, 0]], op: txvalue } { stack: [v], cstack: [[m, 2], [a, 1]], op: push } { stack: [v, 25], cstack: [[m, 2], [a, 3]], op: push } { stack: [v, 25, 10], cstack: [[m, 2], [a, 5]], op: push } { stack: [v, 25, 10, 18], cstack: [[m, 2], [a, 7]], op: exp } { stack: [v, 25, 10^18], cstack: [[m, 2], [a, 8]], op: mul } { stack: [v, 25*10^18], cstack: [[m, 2], [a, 9]], op: lt } { stack: [0], cstack: [[m, 2], [a, 10]], op: null } { stack: [0], cstack: [[m, 2]], op: if } { stack: [0], cstack: [[m, 5], [y, 0]], op: null } { stack: [0], cstack: [[m, 5]], op: run } { stack: [], cstack: [[m, 7], [b, 0]], op: push } { stack: [0], cstack: [[m, 7], [b, 2]], op: txdata } { stack: [x], cstack: [[m, 7], [b, 3]], op: sload } { stack: [0], cstack: [[m, 7], [b, 4]], op: not } { stack: [1], cstack: [[m, 7], [b, 5]], op: push } { stack: [1, 0], cstack: [[m, 7], [b, 7]], op: txdata } { stack: [1, x], cstack: [[m, 7], [b, 8]], op: push } { stack: [1, x, 100], cstack: [[m, 7], [b, 10]], op: lt } { stack: [1, 0], cstack: [[m, 7], [b, 11]], op: not } { stack: [1, 1], cstack: [[m, 7], [b, 12]], op: mul } { stack: [1], cstack: [[m, 7], [b, 13]], op: not } { stack: [1], cstack: [[m, 7], [b, 14]], op: null } { stack: [0], cstack: [[m, 7]], op: if } { stack: [0], cstack: [[m, 9], [y, 0]], op: null } { stack: [], cstack: [[m, 10]], op: run } { stack: [], cstack: [[m, 12], [c, 0]], op: push } { stack: [1], cstack: [[m, 12], [c, 2]], op: txdata } { stack: [y], cstack: [[m, 12], [c, 3]], op: push } { stack: [y,0], cstack: [[m, 12], [c, 5]], op: txdata } { stack: [y,x], cstack: [[m, 12], [c, 6]], op: sstore } { stack: [], cstack: [[m, 12], [c, 7]], op: null } { stack: [], cstack: [[m, 12]], op: null } { stack: [], cstack: [], op: null } and that’s all there is to it. cumbersome to read, but actually quite easy to implement in any statically or dynamically types programming language or perhaps even ultimately in an asic. optimizations in the above design, there is still one major area where optimizations can be made: making the references compact. what the clear and simple style of the above contract hid is that those pointers to a, b, c, m and z aren’t just compact single letters; they are 20-byte hashes. from an efficiency standpoint, what we just did is thus actually substantially worse than what we had before, at least from the point of view of special cases where code is not nearly-duplicated millions of times. also, there is still no incentive for people writing contracts to write their code in such a way that other programmers later on can optimize; if i wanted to code the above in a way that would minimize fees, i would just put a, b and c into the contract directly rather than separating them out into functions. there are two possible solutions: instead of using h(x) = sha3(rlp.encode(x))[12:], use h(x) = sha3(rlp.encode(x))[12:] if len(rlp.encode(x)) >= 20 else x. to summarize, if something is less than 20 bytes long, we include it directly. a concept of “libraries”. the idea behind libraries is that a group of a few scripts can be published together, in a format [ [ ... code ... ], [ ... code ... ], ... ], and these scripts can internally refer to each other with their indices in the list alone. this completely alleviates the problem, but at some cost of harming deduplication, since sub-codes may need to be stored twice. some intelligent thought into exactly how to improve on this concept to provide both deduplication and reference efficiency will be required; perhaps one solution would be for the library to store a list of hashes, and then for the continuation stack to store [ lib, libindex, codeindex ] instead of [ hash, index ]. other optimizations are likely possible. for example, one important weakness of the design described above is that it does not support recursion, offering only while loops to provide turing-completeness. it might seem to, since you can call any function, but if you try to actually try to implement recursion in es2 as described above you soon notice that implementing recursion would require finding the fixed point of an iterated hash (ie. finding x such that h(a + h( c + ... h(x) ... + d) + b) = x), a problem which is generally assumed to be cryptographically impossible. the “library” concept described above does actually fix this at least internally to one library; ideally, a more perfect solution would exist, although it is not necessary. finally, some research should go into the question of making functions first-class; this basically means changing the if and runopcode to pull the destination from the stack rather than from fixed code. this may be a major usability improvement, since you can then code higher-order functions that take functions as arguments like map, but it may also be harmful from an optimization standpoint since code becomes harder to analyze and determine whether or not a given computation is purely functional. fees finally, there is one last question to be resolved. the primary purposes of es2 as described above are twofold: deduplication and optimization. however, optimizations by themselves are not enough; in order for people to actually benefit from the optimizations, and to be incentivized to code in patterns that are optimization-friendly, we need to have a fee structure that supports this. from a deduplication perspective, we already have this; if you are the second person to create a namecoin-like contract, and you want to use a, you can just link to a without paying the fee to instantiate it yourself. however, from an optimization perspective, we are far from done. if we create sha3 in es, and then have the interpreter intelligently replace it with a contract, then the interpreter does get much faster, but the person using sha3 still needs to pay thousands of basefees. thus, we need a mechanism for reducing the fee of specific computations that have been heavily optimized. our current strategy with fees is to have miners or ether holders vote on the basefee, and in theory this system can easily be expanded to include the option to vote on reduced fees for specific scripts. however, this does need to be done intelligently. for example, exp can be replaced with a contract of the following form: push 1 swapn 3 swap while ( dup push 2 mod if ( dupn 2 ) ( push 1 ) dupn 4 mul swapn 4 pop 2 div swap dup mul swap ) pop however, the runtime of this contract depends on the exponent – with an exponent in the range [4,7] the while loop runs three times, in the range [1024, 2047] the while loop runs eleven times, and in the range [2^255, 2^256-1] it runs 256 times. thus, it would be highly dangerous to have a mechanism which can be used to simply set a fixed fee for any contract, since that can be exploited to, say, impose a fixed fee for a contract computing the ackermann function (a function notorious in the world of mathematics because the cost of computing or writing down its output grows so fast that with inputs as low as 5 it becomes larger than the size of the universe). thus, a percentage discount system, where some contracts can enjoy half as large a basefee, may make more sense. ultimately, however, a contract cannot be optimized down to below the cost of calling the optimized code, so we may want to have a fixed fee component. a compromise approach might be to have a discount system, but combined with a rule that no contract can have its fee reduced below 20x the basefee. so how would fee voting work? one approach would be to store the discount of a code item along side that code item’s code, as a number from 1 to 232, where 232 represents no discount at all and 1 represents the highest discounting level of 4294967296x (it may be prudent to set the maximum at 65536x instead for safety). miners would be authorized to make special “discount transactions” changing the discounting number of any code item by a maximum of 1/65536x of its previous value. with such a system, it would take about 40000 blocks or about one month to halve the fee of any given script, a sufficient level of friction to prevent mining attacks and give everyone a chance to upgrade to new clients with more advanced optimizers while still making it possible to update fees as required to ensure future-compatibility. note that the above description is not clean, and is still very much not fleshed out; a lot of care will need to be made in making it maximally elegant and easy to implement. an important point is that optimizers will likely end up replacing entire swaths of es2 code blocks with more efficient machine code, but under the system described above will still need to pay attention to es2 code blocks in order to determine what the fee is. one solution is to have a miner policy offering discounts only to contracts which maintain exactly the same fee when run regardless of their input; perhaps other solutions exist as well. however, one thing is clear: the problem is not an easy one. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum in practice part 3: how to build your own transparent bank on the blockchain | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum in practice part 3: how to build your own transparent bank on the blockchain posted by alex van de sande on december 7, 2015 research & development this is the third and final post on a series on how to use the ethereum wallet to create your own autonomous organisations. on the first post we detailed how to create a token, and on the second we shown how to generate a digital democracy controlled by these tokens. now we will do the full circle and create a token controlled by the organisation! we are going to modify the token contract to allow it to be minted by your dao. so save the address of your current dao in a note pad (pay attention to the icon) and grab this source code and you know the drill: contracts > deploy new contract > solidity source code > pick contract you can fill the parameters any way you want (yes, emojis are permitted on the string fields) but you'll notice one new field that didn't exist before: central minter. here add the address of your newly created democracy contract.   click deploy and let's wait for the transaction to be picked up. after it has at least two confirmations, go to your democracy contract and you'll notice that now it owns a million of your new coins. now if you go to the contracts tab you'll see that there is a new dao dollar (admin page) contract on your collection. select the "minttoken" function to your right and then put any address you own as the "target", and then the amount of new mints you want to create from thin air in their account. press "execute" but don't press send! you'll notice that there is a warning saying that the transaction can't be executed. this happens because only the minter (which is currently set to the dao address) can call that function and you are calling it with your main account. but the calling code is the same, which is why you can simply copy it. instead, copy the contract execution code from the "data" field and put it aside on a notepad. also get the address of your new "mint" contract and save it somewhere. now go back to the democracy contract and create a new proposal with these parameters: as the beneficiary, put the address of your new token leave etheramount blank on the jobdescription just write a small description that you are minting new coins on the transactionbytecode, paste the bytecode you saved from the data field on the previous step     in a few seconds you should be able to see that the details on the proposal. unlike the other fields, transactionbytecode can be extremely lengthy and therefore expensive to store on the blockchain. so instead of archiving it, the person executing the call later will provide the bytecode. but that of course creates a security hole: how can a proposal be voted without the actual code being there? and what prevents a user from executing a different code after the proposal has been voted on? that's why we keep the hash of the bytecode. scroll a bit on the "read from contract" function list and you'll see a proposal checker function, where anyone can put all the function parameters and check if they match the one being voted on. this also guarantees that proposals don't get executed unless the hash of the bytecode matches exactly the one on the provided code.   it's an older code, but it checks out   now everyone can vote on the proposal and after the voting period has passed, anyone with the correct bytecode can ask the votes to be tallied up and the contract to be executed. if the proposal has enough support then the newly minted coins should appear on alice's account, as if it was a transfer from address zero.   why a transfer from address zero? because doing the opposite, sending a coin to 0x00 is a way to effectively destroy it, but more importantly, because it says so on the contract code. you can change that as you prefer.   and now you have a central minter contract that exists solely on the blockchain, completelly fraud-proof as all their activities are logged transparently. the mint can also take coins from circulation by simply sending the coins it has to address zero, or by freezing the funds on any account, but it's mathematically impossible for the mint to do any of those actions or generate more coins without the support of enough shareholders of the mint. possible uses of this dao: the creation of a universal stable crypto currency. by controlling the total amount of coins in circulation the mint shareholders can attempt to create an asset whose value doesn't fluctuate too wildly. issuance of certificates of backed assets: the coins can represent an external currency or items that the mint owns and can prove to it's shareholders and token holders. when the mint acquires or sells more of these assets it can burn or generate more assets to guarantee that their digital inventory will always match their real counterpart digitally backed assets. the mint can hold ether or other ethereum based digital currencies and use that to back the value of the currencies circulating improvements suggestions there are multiple ways that this structure can be yet improved, but we will leave it as an exercise and challenge to the reader: right now votes are made by shareholders based on freely tradable tokens. can instead membership be based on invitation, each member getting a single vote (or maybe use quadratic voting or liquid democracy)? what about other voting mechanisms? maybe the vote instead of being a boolean could be a more flexible arrangement: you could vote to postpone the decision, or you can make a vote that is neutral but still count to the quorum currently all proposals have the same debating period. can you make that proportional to the value transfer being proposed? how would you calculate that to tokens? can you create a better token that can be automatically created by sending ether into it, which can then be retrieved by burning the token, at a fluctuating market price? what else can the dao own or do, besides tokens? previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements olympic rewards announced | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search olympic rewards announced posted by vitalik buterin on august 26, 2015 protocol announcements eth dev would like to thank everyone for their participation in the olympic test network, which helped us greatly in stress testing, optimization, and finding bugs in the ethereum clients, as well as determining what the limits of the current ethereum system are. to that end, we are pleased to announce the olympic reward recipients. on top of the rewards announced below, every miner who mined a block in the olympic chain with block number in the range [310000, 589999] is entitled to receive 0.5 eth per block, and every miner who mined a block in the olympic chain with block number in the range [590000, 999999] is entitled to receive 0.05 eth per block. top miners for blocks 300000-1000000 to claim these rewards, send a transaction on olympic from the address that earned the reward to your desired address. 753f49fb35f0f6f8a7bff0413e376af88f7ff684 (57744 blocks) 2500 eth 891f82ee3e031c5dd85e63b23521b4b9f10c67ff (39526 blocks) 1250 eth 24365bc872e4166081611bb0b5483526742c6616 (39358 blocks) 625 eth 8ef297b17b65ded8142dbdb106818cd09899a8f5 (34142 blocks) 375 eth 027b1495165fec8cfc49bdb8450ded4cbce4b12e (33305 blocks) 250 eth top senders for blocks 300000-1000000 b4e64290541cbf36159e727dfd8d873f77b42149 (5429042 txs) 2500 eth 00fc572550f3bdfe84f72e3eaa99d02a43f69733 (1912815 txs) 1250 eth 8464ba8c33260c70c9ab03487bee8a0dda35d383 (947176 txs) 625 eth 082d4cdf07f386ffa9258f52a5c49db4ac321ec6 (837307 txs) 375 eth 970f4f9eb29e8831514a3ff9b8eb2bb860d4731c (831275 txs) 250 eth lowest address at block 848081 only addresses with nonzero nonce (ie. that sent at least one transaction) were included. 0000000000912d1d5757aa9df23dd770b0195a45 750 eth 00000000097cd8a7d004985ab03af70fe77aa8c2 500 eth 0000000103026f36d9f2ba6468d2816cd5dce83a 250 eth contract rewards to claim these, please send a transaction in olympic specifying your new address to send funds to with the account that created the contract. please place the nonce which the account had when it created the contract in the transaction data (eg. if the nonce is 46, put one byte 0x2e as the data). most storage keys at block 848081: 332b656504f4eabb44c8617a42af37461a34e9dc (1727960 slots) 2500 eth f558470e1ed9e33f2ebf994938722d7f287221f2 (685670 slots) 1250 eth 728d902f5b805a3f49707f36957d876dd6d40e53 (657813 slots) 625 eth 2ea6a457c468ed38b222a2c504a674905b5fcb7e (607504 slots) 375 eth b56e8f043e529bca0344a96cbec8b00bc85c7ca6 (135250 slots) 250 eth contract with highest nonce at block 848081: e35e196a53cabeb4eb4cdd66467427a798d8c24f (4823) 750 eth 3673cb526c8505002e7ba558140b0c4d60009ac9 (2907) 500 eth 66314103c204504486b6ce587109853f4e99e29b (385) 250 eth manually issued rewards for all-around good behavior: phistr90 3500 eth samuel lavery 1750 eth dino mark 1250 eth kobi gurk 500 eth a file containing the amount of eth to be received by each miner can be found here: http://vitalik.ca/files/olympic_miners.txt. to claim these rewards, follow the same process as for top miner/sender rewards described above. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements bootstrapping a decentralized autonomous corporation: part i | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search bootstrapping a decentralized autonomous corporation: part i posted by vitalik buterin on december 31, 2013 research & development corporations, us presidential candidate mitt romney reminds us, are people. whether or not you agree with the conclusions that his partisans draw from that claim, the statement certainly carries a large amount of truth. what is a corporation, after all, but a certain group of people working together under a set of specific rules? when a corporation owns property, what that really means is that there is a legal contract stating that the property can only be used for certain purposes under the control of those people who are currently its board of directors – a designation itself modifiable by a particular set of shareholder. if a corporation does something, it’s because its board of directors has agreed that it should be done. if a corporation hires employees, it means that the employees are agreeing to provide services to the corporation’s customers under a particular set of rules, particularly involving payment. when a corporation has limited liability, it means that specific people have been granted extra privileges to act with reduced fear of legal prosecution by the government – a group of people with more rights than ordinary people acting alone, but ultimately people nonetheless. in any case, it’s nothing more than people and contracts all the way down. however, here a very interesting question arises: do we really need the people? on the one hand, the answer is yes: although in some post-singularity future machines will be able to survive all on their own, for the forseeable future some kind of human action will simply be necessary to interact with the physical world. on the other hand, however, over the past two hundred years the answer has been increasingly no. the industrial revolution allowed us, for the first time, to start replacing human labor with machines on a large scale, and now we have advanced digitized factories and robotic arms that produce complex goods like automobiles all on their own. but this is only automating the bottom; removing the need for rank and file manual laborers, and replacing them with a smaller number of professionals to maintain the robots, while the management of the company remains untouched. the question is, can we approach the problem from the other direction: even if we still need human beings to perform certain specialized tasks, can we remove the management from the equation instead? most companies have some kind of mission statement; often it’s about making money for shareholders; at other times, it includes some moral imperative to do with the particular product that they are creating, and other goals like helping communities sometimes enter the mix, at least in theory. right now, that mission statement exists only insofar as the board of directors, and ultimately the shareholders, interpret it. but what if, with the power of modern information technology, we can encode the mission statement into code; that is, create an inviolable contract that generates revenue, pays people to perform some function, and finds hardware for itself to run on, all without any need for top-down human direction? as let’s talk bitcoin’s daniel larmier pointed out in his own exploration on this concept, in a sense bitcoin itself can be thought of as a very early prototype of exactly such a thing. bitcoin has 21 million shares, and these shares are owned by what can be considered bitcoin’s shareholders. it has employees, and it has a protocol for paying them: 25 btc to one random member of the workforce roughly every ten minutes. it even has its own marketing department, to a large extent made up of the shareholders themselves. however, it is also very limited. it knows almost nothing about the world except for the current time, it has no way of changing any aspect of its function aside from the difficulty, and it does not actually do anything per se; it simply exists, and leaves it up to the world to recognize it. the question is: can we do better? computation the first challenge is obvious: how would such a corporation actually make any decisions? it’s easy to write code that, at least given predictable environments, takes a given input and calculates a desired action to take. but who is going to run the code? if the code simply exists as a computer program on some particular machine, what is stopping the owner of that machine from shutting the whole thing down, or even modifying its code to make it send all of its money to himself? to this problem, there is only one effective answer: distributed computing. however, the kind of distributed computing that we are looking for here is not the same as the distributed computing in projects like seti@home and folding@home; in those cases, there is still a central server collecting data from the distributed nodes and sending out requests. here, rather, we need the kind of distributed computing that we see in bitcoin: a set of rules that decentrally self-validates its own computation. in bitcoin, this is accomplished by a simple majority vote: if you are not helping to compute the blockchain with the majority network power, your blocks will get discarded and you will get no block reward. the theory is that no single attacker will have enough computer power to subvert this mechanism, so the only viable strategy is essentially to “go with the flow” and act honestly to help support the network and receive one’s block reward. so can we simply apply this mechanism to decentralized computation? that is, can we simply ask every computer in the network to evaluate a program, and then reward only those whose answer matches the majority vote? the answer is, unfortunately, no. bitcoin is a special case because bitcoin is simple: it is just a currency, carrying no property or private data of its own. a virtual corporation, on the other hand, would likely need to store the private key to its bitcoin wallet – a piece of data which should be available in its entirety to no one, not to everyone in the way that bitcoin transactions are. but, of course, the private key must still be usable. thus, what we need is some system of signing transactions, and even generating bitcoin addresses, that can be computed in a decentralized way. fortunately, bitcoin allows us to do exactly that. the first solution that might immediately come to mind is multisignature addresses; given a set of a thousand computers that can be relied upon to probably continue supporting the corporations, have each of them create a private key, and generate a 501-of-1000 multisignature address between them. to spend the funds, simply construct a transaction with signatures from any 501 nodes and broadcast it into the blockchain. the problem here is obvious: the transaction would be too large. each signature makes up about seventy bytes, so 501 of them would make a 35 kb transaction – which is very difficult to get accepted into the network as bitcoind by default refuses transactions with any script above 10,000 bytes. second, the solution is specific to bitcoin; if the corporation wants to store private data for non-financial purposes, multisignature scripts are useless. multisignature addresses work because there is a bitcoin network evaluating them, and placing transactions into the blockchain depending on whether or not the evaluation succeeds. in the case of private data, an analogous solution would essentially require some decentralized authority to store the data and give it out only if a request has 501 out of 1000 signatures as needed – putting us right back where we started. however, there is still hope in another solution; the general name given to this by cryptographers is “secure multiparty computation”. in secure multiparty computation, the inputs to a program (or, more precisely, the inputs to a simulated “circuit”, as secure multiparty computation cannot handle “if” statements and conditional looping) are split up using an algorithm calledshamir’s secret sharing, and a piece of the information is given to each participant. shamir’s secret sharing can be used to split up any data into n pieces such that any k of them, but no k-1 of them, are sufficient to recover the original data – you choose what k and n are when running the algorithm. 2-of-3, 5-of-10 and 501-of-1000 are all possible. a circuit can then be evaluated on the pieces of data in a decentralized way, such that at the end of the computation everyone has a piece of the result of the computation, but at no point during the computation does any single individual get even the slightest glimpse of what is going on. finally, the pieces are put together to reveal the result. the runtime of the algorithm is o(n3), meaning that the number of computational steps that it takes to evaluate a computation is roughly proportional to the cube of the number of participants; at 10 nodes, 1000 computational steps, and at 1000 nodes 1 billion steps. a simple billion-step loop in c++ takes about twenty seconds on my own laptop, and servers can do it in a fraction of a second, so 1000 nodes is currently roughly at the limit of computational practicality. as it turns out, secure multiparty computation can be used to generate bitcoin addresses and sign transactions. for address generation, the protocol is simple: everyone generates a random number as a private key. everyone calculates the public key corresponding to the private key. everyone reveals their public key, and uses shamir’s secret sharing algorithm to calculate a public key that can be reconstructed from any 501 of the thousand public keys revealed. an address is generated from that public key. because public keys can be added, subtracted , multiplied and even divided by integers, surprisingly this algorithm works exactly as you would expect. if everyone were to then put together a 501-of-1000 private key in the same way, that private key would be able to spend the money sent to the address generated by applying the 501-of-1000 algorithm to the corresponding public keys. this works because shamir’s secret sharing is really just an algebraic formula – that is to say, it uses only addition, subtraction, multiplication and division, and one can compute this formula “over” public keys just as easily as with addresses; as a result, it doesn’t matter if the private key to public key conversion is done before the algebra or after it. signing transactions can be done in a similar way, although the process is somewhat more complicated. the beauty of secure multiparty computation is that it extends beyond just bitcoin; it can just as easily be used to run the artificial intelligence algorithm that the corporation relies on to operate. so-called “machine learning”, the common name for a set of algorithms that detect patterns in real-world data and allow computers to model it without human intervention and are employed heavily in fields like spam filters and self-driving cars, is also “just algebra”, and can be implemented in secure multiparty computation as well. really, any computation can, if that computation is broken down into a circuit on the input’s individual bits. there is naturally some limit to the complexity that is possible; converting complex algorithms into circuits often introduces additional complexity, and, as described above, shamir’s secret sharing can get expensive all by itself. thus, it should only really be used to implement the “core” of the algorithm; more complex high-level thinking tasks are best resolved by outside contractors. excited about this topic? look forward to parts 2, 3 and 4: how decentralized corporations can interact with the outside world, how some simple secure multiparty computation circuits work on a mathematical level, and two examples of how these decentralized corporations can make a difference in the real world. see also: http://letstalkbitcoin.com/is-bitcoin-overpaying-for-false-security/ http://bitcoinmagazine.com/7119/bootstrapping-an-autonomous-decentralized-corporation-part-2-interacting-with-the-world/ http://bitcoinmagazine.com/7235/bootstrapping-a-decentralized-autonomous-corporation-part-3-identity-corp/ previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements on abstraction | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on abstraction posted by vitalik buterin on july 5, 2015 research & development special thanks to gavin wood, vlad zamfir, our security auditors and others for some of the thoughts that led to the conclusions described in this post one of ethereum's goals from the start, and arguably its entire raison d'être, is the high degree of abstraction that the platform offers. rather than limiting users to a specific set of transaction types and applications, the platform allows anyone to create any kind of blockchain application by writing a script and uploading it to the ethereum blockchain. this gives an ethereum a degree of future-proof-ness and neutrality much greater than that of other blockchain protocols: even if society decides that blockchains aren't really all that useful for finance at all, and are only really interesting for supply chain tracking, self-owning cars and self-refilling dishwashers and playing chess for money in a trust-free form, ethereum will still be useful. however, there still are a substantial number of ways in which ethereum is not nearly as abstract as it could be. cryptography currently, ethereum transactions are all signed using the ecdsa algorithm, and specifically bitcoin's secp256k1 curve. elliptic curve signatures are a popular kind of signature today, particularly because of the smaller signature and key sizes compared to rsa: an elliptic curve signature takes only 65 bytes, compared to several hundred bytes for an rsa signature. however, it is becoming increasingly understood that the specific kind of signature used by bitcoin is far from optimal; ed25519 is increasingly recognized as a superior alternative particularly because of its simpler implementation, greater hardness against side-channel attacks and faster verification. and if quantum computers come around, we will likely have to move to lamport signatures. one suggestion that some of our security auditors, and others, have given us is to allow ed25519 signatures as an option in 1.1. but what if we can stay true to our spirit of abstraction and go a bit further: let people use whatever cryptographic verification algorithm that they want? is that even possible to do securely? well, we have the ethereum virtual machine, so we have a way of letting people implement arbitrary cryptographic verification algorithms, but we still need to figure out how it can fit in. here is a possible approach: every account that is not a contract has a piece of "verification code" attached to it. when a transaction is sent, it must now explicitly specify both sender and recipient. the first step in processing a transaction is to call the verification code, using the transaction's signature (now a plain byte array) as input. if the verification code outputs anything nonempty within 50000 gas, the transaction is valid. if it outputs an empty array (ie. exactly zero bytes; a single \x00 byte does not count) or exits with an exception condition, then it is not valid. to allow people without eth to create accounts, we implement a protocol such that one can generate verification code offline and use the hash of the verification code as an address. people can send funds to that address. the first time you send a transaction from that account, you need to provide the verification code in a separate field (we can perhaps overload the nonce for this, since in all cases where this happens the nonce would be zero in any case) and the protocol (i) checks that the verification code is correct, and (ii) swaps it in (this is roughly equivalent to "pay-to-script-hash" in bitcoin). this approach has a few benefits. first, it does not specify anything about the cryptographic algorithm used or the signature format, except that it must take up at most 50000 gas (this value can be adjusted up or down over time). second, it still keeps the property of the existing system that no pre-registration is required. third, and quite importantly, it allows people to add higher-level validity conditions that depend on state: for example, making transactions that spend more gavcoin than you currently have actually fail instead of just going into the blockchain and having no effect. however, there are substantial changes to the virtual machine that would have to be made for this to work well. the current virtual machine is designed well for dealing with 256-bit numbers, capturing the hashes and elliptic curve signatures that are used right now, but is suboptimal for algorithms that have different sizes. additionally, no matter how well-designed the vm is right now, it fundamentally adds a layer of abstraction between the code and the machine. hence, if this will be one of the uses of the vm going forward, an architecture that maps vm code directly to machine code, applying transformations in the middle to translate specialized opcodes and ensure security, will likely be optimal particularly for expensive and exotic cryptographic algorithms like zk-snarks. and even then, one must take care to minimize any "startup costs" of the virtual machine in order to further increase efficiency as well as denial-of-service vulnerability; together with this, a gas cost rule that encourages re-using existing code and heavily penalizes using different code for every account, allowing just-in-time-compiling virtual machines to maintain a cache, may also be a further improvement. the trie perhaps the most important data structure in ethereum is the patricia tree. the patricia tree is a data structure that, like the standard binary merkle tree, allows any piece of data inside the trie to be securely authenticated against a root hash using a logarithmically sized (ie. relatively short) hash chain, but also has the important property that data can be added, removed or modified in the tree extremely quickly, only making a small number of changes to the entire structure. the trie is used in ethereum to store transactions, receipts, accounts and particularly importantly the storage of each account. one of the often cited weaknesses of this approach is that the trie is one particular data structure, optimized for a particular set of use cases, but in many cases accounts will do better with a different model. the most common request is a heap: a data structure to which elements can quickly be added with a priority value, and from which the lowest-priority element can always be quickly removed particularly useful in implementations of markets with bid/ask offers. right now, the only way to do this is a rather inefficient workaround: write an implementation of a heap in solidity or serpent on top of the trie. this essentially means that every update to the heap requires a logarithmic number of updates (eg. at 1000 elements, ten updates, at 1000000 elements, twenty updates) to the trie, and each update to the trie requires changes to a logarithmic number (once again ten at 1000 elements and twenty at 1000000 elements) of items, and each one of those requires a change to the leveldb database which uses a logarithmic-time-updateable trie internally. if contracts had the option to have a heap instead, as a direct protocol feature, then this overhead could be cut down substantially. one option to solve this problem is the direct one: just have an option for contracts to have either a regular trie or a heap, and be done with it. a seemingly nicer solution, however, is to generalize even further. the solution here is as follows. rather than having a trie or a treap, we simply have an abstract hash tree: there is a root node, which may be empty or which may be the hash of one or more children, and each child in turn may either be a terminal value or the hash of some set of children of its own. an extension may be to allow nodes to have both a value and children. this would all be encoded in rlp; for example, we may stipulate that all nodes must be of the form: [val, child1, child2, child3....] where val must be a string of bytes (we can restrict it to 32 if desired), and each child (of which there can be zero or more) must be the 32 byte sha3 hash of some other node. now, we have the virtual machine's execution environment keep track of a "current node" pointer, and add a few opcodes: getval: pushes the value of the node at the current pointer onto the stack setval: sets the value at the of the node at the current pointer to the value at the top of the stack getchildcount: gets the number of children of the node addchild: adds a new child node (starting with zero children of its own) removechild: pops off a child node descend: descend to the kth child of the current node (taking k as an argument from the stack) ascend: ascend to the parent ascendroot: ascend to the root node accessing a merkle tree with 128 elements would thus look like this: def access(i): ~ascendroot() return _access(i, 7) def _access(i, depth): while depth > 0: ~descend(i % 2) i /= 2 depth -= 1 return ~getval() creating the tree would look like this: def create(vals): ~ascendroot() while ~getchildcount() > 0: ~removechild() _create(vals, 7) def _create(vals:arr, depth): if depth > 0: # recursively create left child ~addchild() ~descend(0) _create(slice(vals, 0, 2**(depth 1)), depth 1) ~ascend() # recursively create right child ~addchild() ~descend(1) _create(slice(vals, 2**(depth 1), 2**depth), depth 1) ~ascend() else: ~setval(vals[0]) clearly, the trie, the treap and in fact any other tree-like data structure could thus be implemented as a library on top of these methods. what is particularly interesting is that each individual opcode is constant-time: theoretically, each node can keep track of the pointers to its children and parent on the database level, requiring only one level of overhead. however, this approach also comes with flaws. particularly, note that if we lose control of the structure of the tree, then we lose the ability to make optimizations. right now, most ethereum clients, including c++, go and python, have a higher-level cache that allows updates to and reads from storage to happen in constant time if there are multiple reads and writes within one transaction execution. if tries become de-standardized, then optimizations like these become impossible. additionally, each individual trie structure would need to come up with its own gas costs and its own mechanisms for ensuring that the tree cannot be exploited: quite a hard problem, given that even our own trie had a medium level of vulnerability until recently when we replaced the trie keys with the sha3 hash of the key rather than the actual key. hence, it's unclear whether going this far is worth it. currency it's well-known and established that an open blockchain requires some kind of cryptocurrency in order to incentivize people to participate in the consensus process; this is the kernel of truth behind this otherwise rather silly meme: however, can we create a blockchain that does not rely on any specific currency, instead allowing people to transact using whatever currency they wish? in a proof of work context, particularly a fees-only one, this is actually relatively easy to do for a simple currency blockchain; just have a block size limit and leave it to miners and transaction senders themselves to come to some equilibrium over the transaction price (the transaction fees may well be done as a batch payment via credit card). for ethereum, however, it is slightly more complicated. the reason is that ethereum 1.0, as it stands, comes with a built-in gas mechanism which allows miners to safely accept transactions without fear of being hit by denial-of-service attacks; the mechanism works as follows: every transaction specifies a max gas count and a fee to pay per unit gas. suppose that the transaction allows itself a gas limit of n. if the transaction is valid, and takes less than n computational steps (say, m computational steps), then it pays m steps worth of the fee. if the transaction consumes all n computational steps before finishing, the execution is reverted but it still pays n steps worth of the fee. this mechanism relies on the existence of a specific currency, eth, which is controlled by the protocol. can we replicate it without relying on any one particular currency? as it turns out, the answer is yes, at least if we combine it with the "use any cryptography you want" scheme above. the approach is as follows. first, we extend the above cryptography-neutrality scheme a bit further: rather than having a separate concept of "verification code" to decide whether or not a particular transaction is valid, simply state that there is only one type of account a contract, and a transaction is simply a message coming in from the zero address. if the transaction exits with an exceptional condition within 50000 gas, the transaction is invalid; otherwise it is valid and accepted. within this model, we then set up accounts to have the following code: check if the transaction is correct. if not, exit. if it is, send some payment for gas to a master contract that will later pay the miner. send the actual message. send a message to ping the master contract. the master contract then checks how much gas is left, and refunds a fee corresponding to the remaining amount to the sender and sends the rest to the miner. step 1 can be crafted in a standardized form, so that it clearly consumes less than 50000 gas. step 3 can similarly be constructed. step 2 can then have the message provide a gas limit equal to the transaction's specified gas limit minus 100000. miners can then pattern-match to only accept transactions that are of this standard form (new standard forms can of course be introduced over time), and they can be sure that no single transaction will cheat them out of more than 50000 steps of computational energy. hence, everything becomes enforced entirely by the gas limit, and miners and transaction senders can use whatever currency they want. one challenge that arises is: how do you pay contracts? currently, contracts have the ability to "charge" for services, using code like this registry example: def reserve(_name:bytes32): if msg.value > 100 * 10**18: if not self.domains[_name].owner: self.domains[_name].owner = msg.sender with a sub-currency, there is no such clear mechanism of tying together a message and a payment for that message. however, there are two general patterns that can act as a substitute. the first is a kind of "receipt" interface: when you send a currency payment to someone, you have the ability to ask the contract to store the sender and value of the transaction. something like registrar.reserve("blahblahblah.eth") would thus be replaced by: gavcoin.sendwithreceipt(registrar, 100 * 10**18) registrar.reserve("blahblahblah.eth") the currency would have code that looks something like this: def sendwithreceipt(to, value): if self.balances[msg.sender] >= value: self.balances[msg.sender] -= value self.balances[to] += value self.last_sender = msg.sender self.last_recipient = to self.last_value = value def getlastreceipt(): return([self.last_sender, self.last_recipient, self.value]:arr) and the registrar would work like this: def reserve(_name:bytes32): r = gavcoin.getlastreceipt(outitems=3) if r[0] == msg.sender and r[1] == self and r[2] >= 100 * 10**18: if not self.domains[_name].owner: self.domains[_name].owner = msg.sender essentially, the registrar would check the last payment made in that currency contract, and make sure that it is a payment to itself. in order to prevent double-use of a payment, it may make sense to have the get_last_receipt method destroy the receipt in the process of reading it. the other pattern is to have a currency have an interface for allowing another address to make withdrawals from your account. the code would then look as follows on the caller side: first, approve a one-time withdrawal of some number of currency units, then reserve, and the reservation contract attempts to make the withdrawal and only goes forward if the withdrawal succeeds: gavcoin.approveonce(registrar, 100) registrar.reserve("blahblahblah.eth") and the registrar would be: def reserve(_name:bytes32): if gavcoin.sendcoinfrom(msg.sender, 100, self) == success: if not self.domains[_name].owner: self.domains[_name].owner = msg.sender the second pattern has been standardized at the standardized contract apis wiki page. currency-agnostic proof of stake the above allows us to create a completely currency-agnostic proof-of-work blockchain. however, to what extent can currency-agnosticism be added to proof of stake? currency-agnostic proof of stake is useful for two reasons. first, it creates a stronger impression of economic neutrality, which makes it more likely to be accepted by existing established groups as it would not be seen as favoring a particular specialized elite (bitcoin holders, ether holders, etc). second, it increases the amount that will be deposited, as individuals holding digital assets other than ether would have a very low personal cost in putting some of those assets into a deposit contract. at first glance, it seems like a hard problem: unlike proof of work, which is fundamentally based on an external and neutral resource, proof of stake is intrinsically based on some kind of currency. so how far can we go? the first step is to try to create a proof of stake system that works using any currency, using some kind of standardized currency interface. the idea is simple: anyone would be able to participate in the system by putting up any currency as a security deposit. some market mechanism would then be used in order to determine the value of each currency, so as to estimate the amount of each currency that would need to be put up in order to obtain a stake depositing slot. a simple first approximation would be to maintain an on-chain decentralized exchange and read price feeds; however, this ignores liquidity and sockpuppet issues (eg. it's easy to create a currency and spread it across a small group of accounts and pretend that it has a value of $1 trillion per unit); hence, a more coarse-grained and direct mechanism is required. to get an idea of what we are looking for, consider david friedman's description of one particular aspect of the ancient athenian legal system: the athenians had a straightforward solution to the problem of producing public goods such as the maintainance of a warship or the organizing of a public festival. if you were one of the richest athenians, every two years you were obligated to produce a public good; the relevant magistrate would tell you which one. "as you doubtless know, we are sending a team to the olympics this year. congratulations, you are the sponsor." or "look at that lovely trireme down at the dock. this year guess who gets to be captain and paymaster." such an obligation was called a liturgy. there were two ways to get out of it. one was to show that you were already doing another liturgy this year or had done one last year. the other was to prove that there was another athenian, richer than you, who had not done one last year and was not doing one this year. this raises an obvious puzzle. how, in a world without accountants, income tax, public records of what people owned and what it was worth, do i prove that you are richer than i am? the answer is not an accountant’s answer but an economist’s—feel free to spend a few minutes trying to figure it out before you turn the page. the solution was simple. i offer to exchange everything i own for everything you own. if you refuse, you have admitted that you are richer than i am, and so you get to do the liturgy that was to be imposed on me. here, we have a rather nifty scheme for preventing people that are rich from pretending that they are poor. now, however, what we are looking for is a scheme for preventing people that are poor from pretending that they are rich (or more precisely, preventing people that are releasing small amounts of value into the proof of stake security deposit scheme from pretending that they are staking a much larger amount). a simple approach would be a swapping scheme like that, but done in reverse via a voting mechanic: in order to join the stakeholder pool, you would need to be approved by 33% of the existing stakeholders, but every stakeholder that approves you would have to face the condition that you can exchange your stake for theirs: a condition that they would not be willing to meet if they thought it likely that the value of your stake actually would drop. stakeholders would then charge an insurance fee for signing stake that is likely to strongly drop against the existing currencies that are used in the stake pool. this scheme as described above has two substantial flaws. first, it naturally leads to currency centralization, as if one currency is dominant it will be most convenient and safe to also stake in that currency. if there are two assets, a and b, the process of joining using currency a, in this scheme, implies receiving an option (in the financial sense of the term) to purchase b at the exchange rate of a:b at the price at the time of joining, and this option would thus naturally have a cost (which can be estimated via the black-scholes model). just joining with currency a would be simpler. however, this can be remedied by asking stakeholders to continually vote on the price of all currencies and assets used in the stake pool an incentivized vote, as the vote reflects both the weight of the asset from the point of view of the system and the exchange rate at which the assets can be forcibly exchanged. a second, more serious flaw, however, is the possibility of pathological metacoins. for example, one can imagine a currency which is backed by gold, but which has the additional rule, imposd by the institution backing it, that forcible transfers initiated by the protocol "do not count"; that is, if such a transfer takes place, the allocation before the transfer is frozen and a new currency is created using that allocation as its starting point. the old currency is no longer backed by gold, and the new one is. athenian forcible-exchange protocols can get you far when you can actually forcibly exchange property, but when one can deliberately create pathological assets that arbitrarily circumvent specific transaction types it gets quite a bit harder. theoretically, the voting mechanism can of course get around this problem: nodes can simply refuse to induct currencies that they know are suspicious, and the default strategy can tend toward conservatism, accepting a very small number of currencies and assets only. altogether, we leave currency-agnostic proof of stake as an open problem; it remains to be seen exactly how far it can go, and the end result may well be some quasi-subjective combination of trustdavis and ripple consensus. sha3 and rlp now, we get to the last few parts of the protocol that we have not yet taken apart: the hash algorithm and the serialization algorithm. here, unfortunately, abstracting things away is much harder, and it is also much harder to tell what the value is. first of all, it is important to note that even though we have shows how we could conceivably abstract away the trees that are used for account storage, it is much harder to see how we could abstract away the trie on the top level that keeps track of the accounts themselves. this tree is necessarily system-wide, and so one can't simply say that different users will have different versions of it. the top-level trie relies on sha3, so some kind of specific hashing algorithm there must stay. even the bottom-level data structures will likely have to stay sha3, since otherwise there would be a risk of a hash function being used that is not collision-resistant, making the whole thing no longer strongly cryptographically authenticated and perhaps leading to forks between full clients and light clients. rlp is similarly unavoiable; at the very least, each account needs to have code and storage, and the two need to be stored together some how, and that is already a serialization format. fortunately, however, sha3 and rlp are perhaps the most well-tested, future-proof and robust parts of the protocol, so the benefit from switching to something else is quite small. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements jeff’s ethereum ðξv update i | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search jeff’s ethereum ðξv update i posted by jeffrey wilcke on november 2, 2014 research & development i thought it was about time i'd give an update on my side of things for those interested in knowing how we're doing on the dutch side. my name is jeff, a founder of ethereum and one of the three directors (alongside vitalik and gavin) of ethereum ðξv, the development entity building ethereum and all the associated tech. over the past months i've been been looking for a suitable office space to host the amsterdam hub. unfortunately it takes more work than i initially anticipated and have got nothing to show for so far. i've past on this tasks to my good friend and now colleague maran. maran will do all he can to find the best suitable place for the ams hub and the development of mist. those who are using the ethereum programming language mutan you may want to switch over to serpent or the soon to be de-facto programming language solidity. i've made the sensible decision to focus my attention on to more pressing matters such as the development of the protocol and the browser. perhaps in the future i'll have time to pick up it's development again. ðξv amsterdam the lawyers have finally, after 2 months, gotten around to set up the company here in amsterdam (ugh, the dutch and their bureaucracy eh) and we've found a bank that is willing to accept us as their loyal customers (...). at the moment we have a few options for our office space and i'll write about them as soon as i know something more concrete. the team it's about time the ams team got a proper introduction. these guys do some serious good work! the first that joined the ams team is alex van de sande (aka avsa). alex is a gifted ux engineer and he's been with us for quite a while. it was only a matter of time before he became an official member of the ðξv team. alex has taken up the task of ui development and ux expert and is prototyping the latest of the web3 browser. the second that joined the team is viktor trón. vik is a crazy math-head and is currently hacking away at the new devp2p and testing it rigourously. i've known vik all the way back since the start of the project somewhere in jan/feb, he's a great guy and a real asset to this team. the third that joined the team is felix lange. felix is a die-hard gopher (yay!) and the first thing he pointed out to me was that i had done a bad job looking after my go routines and there were a lot of race conditions, so nice of him (-; felix is going to work on the whisper implementation once the spec has been formally finalised. felix is a super star gopher and has the ability to become a true ethereum core dev. the fourth that joined the team is daniel nagy. daniel has a history in crypto and security and his first tasks is to create a comprehensive spec for our dht implementation and the development thereof. last but certainly not least is maran hidskes. maran has been on this team before but in a completely different role. maran used to work on the protocol but after spawning a crying, -peeing, -pooping machine new member of his family he decided to take some time off. now his main role is to look after the daunting task; the administration of ðξv amsterdam. even though they are not on anyone's team, i like to thank nick, caktux and joris for their ongoing effort in developing out our build systems. i'd also like to thank nick specifically for pointing out the inconsistencies between our implementations: nick, you truly are a great pain in my ass (-; onwards while we are marching towards the next instalment in the proof of concept (poc-7) we still have got quite some work ahead of us. recently i've started to build a toolset so we may test out christoph (he's on the berlin team) awesome tests suit. christoph has put a tremendous amount of work in developing out a proper testing suit for the ethereum protocol. i never knew people could enjoy writing tests like you do, you've got my uttermost most respect. i've also started a cross-implementation javascript framework called ethereum.js. ethereum.js is quickly gaining adoption from the rest of the ether hackers and is already in use by the go websocket & json rpc implementation, c++ json rpc implementation and the node.js implementation. ethereum.js is a true ðξv cross implementation team effort. our polish partners at imapp (paweł and artur) have completed their first implementation of the jit-compiled llvm-based evm implementation and have agreed to create a go bridge so that mist may benefit from the speed increase in running ethereum contracts mentioned earlier in gav's update. finally, the ux and ui refinement of mist, our flagship consumer product and next-generation browser, continues at great pace. an early 'preview' of the mist interface can be downloaded here. it's going to be impossible to deliver that type of behemoth of a browser for the first version but it will certainly be the end-goal. fin i shall try to keep up writing blog post with updates regarding mist, the protocol and ðξv in general so stay tuned! jeff (jeffrey.spammenot.wilcke@ethdev.com) previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements gav’s ethereum ðξv update iii | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search gav’s ethereum ðξv update iii posted by gavin wood on november 18, 2014 research & development i'm gavin wood, a co-founder of ethereum and, along with vitalik buterin and jeffrey wilcke, one of the three directors of the eth dev, the nfp organisation that is managing the development (under contract from ethereum suisse) of the ethereum blockchain. this is a small update to let you all know what has been going on recently. i sit here on an immaculate couch that has been zapped forward in time from the 1960s. it is in the room that will become the chillout & wind-down room of the heart of the (c++) ethereum development operation. surrounding me is alex leverington on a bond villain's easy chair, and aeron buchanan stuck behind a locker that looks as though it was an original prop from m*a*s*h. lighting equipment from a soviet blade runner, now forgotten except in berlin's coolest districts where renaissance chemistry and 60s luxury breath frankensteinesque life into it, provides an unyielding glow to the work in progress. there is still much to be done here (i feel a little like i'm on the set of challenge anneka) but it is undeniably taking shape. this is thanks mostly to our own anneka rice, sarah o'neill, who is working around the clock to get this place ready for ðξvcon-0, our first developer symposium. helping her, similarly around the clock, is the inimitable roland, a hardened international interior outfitter whose memoirs i can't wait to read. on a personal note, i must say these last few months have been some of the busiest of my life. i spent the last couple of weeks between switzerland and the uk, visiting stephan, ian and louis. despite the draws of the north of england, it's nice to be back in berlin; the combination of great burgers and cocktails, beautiful surroundings and nice people makes it difficult to leave. awesome c++ coders are welcome to take that as a hint: we're still hiring (-: technicals during the last couple of weeks we've made a number of important revisions to the protocol, mostly provisions for creating light-client ðapp nodes. there will be a directors' post in due course detailing these, but suffice it to say we are as committed as ever that the ethereum blockchain make possible the massively multi-user decentralised applications for all sizes of devices. the seventh in our proof-of-concept series is awaiting imminent release and the final in the series, poc-8 will be starting development shortly. fresh meat as time goes on, our team moves from strength to strength. i'm pleased to announce that dr. sven ehlert has joined us. he will be leading development operations; cleaning up the build process, making the build as robust as possible, assisting caktux in our ci systems and, most importantly, helping architect a stress-testing harness in which we'll be simulating a series of extreme situations, measuring and analysing. he's also a scrum aficionado and will be helping us streamline some of our development processes as our team grows. it is with great pleasure i can also announce that dr. jutta steiner will also be working closely with us in the capacity of managing our security audit. as well as being an enthusiastic ðapp-developer, she comes with an excellent track record of handling projects and a superb understanding of not only this cutting edge technology but also the human processes that must go on behind it. i must also shout out to dr. andreas lubbe; though a long-time member of the berlin ethereum community and having worked on ethereum-related code (a notable devotee of node.js), we have recently started working much more closely together on the secure ethereum contract documentation (secdoc) framework and the associated natural language specification format, natspec. i look forward to some great collaboration. aside from lefteris, who began his first official day with us today (working with christian on solidity, and more specifically on the secdoc and natspec portions of it), we have two new developers joining us: yann levreau and arkadiy paronyan. yann, a recent arrival in berlin from his native france will be joined by arkadiy who is travelling all the way from moscow to become part of the team. both have substantial experience in c++ and related technologies and will be helping us flesh out the developer tools and in particular pave the way to the ide vision. finally, i'm happy to report that christoph jentzsch, though originally joining us for only 2 months (while taking time out from his doctoral studies), will be joining the project full time in the new year and continuing his much appreciated work on our tests and the general c++ health and robustness. ðξvcon-0 as time rushes by, sarah, roland and their team rush even more to finish our hub. for they know that come monday the 24th, berlin will have some new arrivals. developers and collaborators from around the globe will descend on 37a waldemarstraße, berlin 10999 for a week of getting everybody on the same page. it is devcon-0: ethereum's first developer symposium. conceived by myself and jeff on a sleepy train from zug to zurich as a means of getting the amsterdam/go guys on the same page as the berlin/c++ guys it has evolved into a showcase, set of seminars and workshops of all of ðξv, our technologies, our personnel and some of our close collaborators. it is a chance for us each to build lasting professional relationships and bond in what will become a project that may if not define, certainly form a hallmark, on our professional lifes. our hub will play host to around 40 people, the vast majority of which are accomplished technical minds that jeff, myself or vitalik has at one point or another mentioned, and for the period of a week we will be chatting, mingling and sharing our ideas, hopes and dreams for everything blockchain, decentralised and disruption related. it's going to be awesome: look out for the videos! ever closer aside from the continuing work towards starting poc-8 and the alpha series, i'm glad to report that the solidity project storms onwards under the stewardship of christian: the first contracts compiled with solidity have been delivered and tested working on the testnet, and as i write this i see another pull request for state mappings. great stuff. alex has also been working tirelessly on our crypto code and is now beginning work on the p2p layer, the full strategy for which we'll be seeing in his address at ðξvcon. marek and marian have also been busy on the javascript api, and i can assure any javascript ðapp developers that they will have a lot to look forward to in poc-8. summing up there are also a few other developments and personnel i'd love to announce, but i fear, once again, it will have to wait until next time. watch this space for a post-ðξvcon update! gav. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements programming society with asm: gavin wood at assembly 2014 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search programming society with asm: gavin wood at assembly 2014 posted by stephan tual on august 6, 2014 research & development previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements on public and private blockchains | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on public and private blockchains posted by vitalik buterin on august 7, 2015 research & development over the last year the concept of "private blockchains" has become very popular in the broader blockchain technology discussion. essentially, instead of having a fully public and uncontrolled network and state machine secured by cryptoeconomics (eg. proof of work, proof of stake), it is also possible to create a system where access permissions are more tightly controlled, with rights to modify or even read the blockchain state restricted to a few users, while still maintaining many kinds of partial guarantees of authenticity and decentralization that blockchains provide. such systems have been a primary focus of interest from financial institutions, and have in part led to a backlash from those who see such developments as either compromising the whole point of decentralization or being a desperate act of dinosaurish middlemen trying to stay relevant (or simply committing the crime of using a blockchain other than bitcoin). however, for those who are in this fight simply because they want to figure out how to best serve humanity, or even pursue the more modest goal of serving their customers, what are the practical differences between the two styles? first, what exactly are the options at hand? to summarize, there are generally three categories of blockchain-like database applications: public blockchains: a public blockchain is a blockchain that anyone in the world can read, anyone in the world can send transactions to and expect to see them included if they are valid, and anyone in the world can participate in the consensus process the process for determining what blocks get added to the chain and what the current state is. as a substitute for centralized or quasi-centralized trust, public blockchains are secured by cryptoeconomics the combination of economic incentives and cryptographic verification using mechanisms such as proof of work or proof of stake, following a general principle that the degree to which someone can have an influence in the consensus process is proportional to the quantity of economic resources that they can bring to bear. these blockchains are generally considered to be "fully decentralized". consortium blockchains: a consortium blockchain is a blockchain where the consensus process is controlled by a pre-selected set of nodes; for example, one might imagine a consortium of 15 financial institutions, each of which operates a node and of which 10 must sign every block in order for the block to be valid. the right to read the blockchain may be public, or restricted to the participants, and there are also hybrid routes such as the root hashes of the blocks being public together with an api that allows members of the public to make a limited number of queries and get back cryptographic proofs of some parts of the blockchain state. these blockchains may be considered "partially decentralized". fully private blockchains: a fully private blockchain is a blockchain where write permissions are kept centralized to one organization. read permissions may be public or restricted to an arbitrary extent. likely applications include database management, auditing, etc internal to a single company, and so public readability may not be necessary in many cases at all, though in other cases public auditability is desired. in general, so far there has been little emphasis on the distinction between consortium blockchains and fully private blockchains, although it is important: the former provides a hybrid between the "low-trust" provided by public blockchains and the "single highly-trusted entity" model of private blockchains, whereas the latter can be more accurately described as a traditional centralized system with a degree of cryptographic auditability attached. however, to some degree there is good reason for the focus on consortium over private: the fundamental value of blockchains in a fully private context, aside from the replicated state machine functionality, is cryptographic authentication, and there is no reason to believe that the optimal format of such authentication provision should consist of a series of hash-linked data packets containing merkle tree roots; generalized zero knowledge proof technology provides a much broader array of exciting possibilities about the kinds of cryptographic assurances that applications can provide their users. in general, i would even argue that generalized zero-knowledge-proofs are, in the corporate financial world, greatly underhyped compared to private blockchains. for now, i will thus focus on the simpler "private versus public" blockchain discussion. in general, the idea that there is "one true way" to be blockchaining is completely wrong headed, and both categories have their own advantages and disadvantages. first, private blockchains. compared to public blockchains, they have a number of advantages: the consortium or company running a private blockchain can easily, if desired, change the rules of a blockchain, revert transactions, modify balances, etc. in some cases, eg. national land registries, this functionality is necessary; there is no way a system would be allowed to exist where dread pirate roberts can have legal ownership rights over a plainly visible piece of land, and so an attempt to create a government-uncontrollable land registry would in practice quickly devolve into one that is not recognized by the government itself. of course, one can argue that one can do this on a public blockchain by giving the government a backdoor key to a contract; the counter-argument to that is that such an approach is essentially a rube goldbergian alternative to the more efficient route of having a private blockchain, although there is in turn a partial counter-argument to that that i will describe later. the validators are known, so any risk of a 51% attack arising from some miner collusion in china does not apply. transactions are cheaper, since they only need to be verified by a few nodes that can be trusted to have very high processing power, and do not need to be verified by ten thousand laptops. this is a hugely important concern right now, as public blockchains tend to have transaction fees exceeding $0.01 per tx, but it is important to note that it may change in the long term with scalable blockchain technology that promises to bring public-blockchain costs down to within one or two orders of magnitude of an optimally efficient private blockchain system nodes can be trusted to be very well-connected, and faults can quickly be fixed by manual intervention, allowing the use of consensus algorithms which offer finality after much shorter block times. improvements in public blockchain technology, such as ethereum 1.0's uncle concept and later proof of stake, can bring public blockchains much closer to the "instant confirmation" ideal (eg. offering total finality after 15 seconds, rather than 99.9999% finality after two hours as does bitcoin), but even still private blockchains will always be faster and the latency difference will never disappear as unfortunately the speed of light does not increase by 2x every two years by moore's law. if read permissions are restricted, private blockchains can provide a greater level of, well, privacy. given all of this, it may seem like private blockchains are unquestionably a better choice for institutions. however, even in an institutional context, public blockchains still have a lot of value, and in fact this value lies to a substantial degree in the philosophical virtues that advocates of public blockchains have been promoting all along, among the chief of which are freedom, neutrality and openness. the advantages of public blockchains generally fall into two major categories: public blockchains provide a way to protect the users of an application from the developers, establishing that there are certain things that even the developers of an application have no authority to do. from a naive standpoint, it may be hard to understand why an application developer would want to voluntarily give up power and hamstring themselves. however, more advanced economic analysis provides two reasons why, in thomas schelling's words, weakness can be a strength. first, if you explicitly make it harder or impossible for yourself to do certain things, then others will be more likely to trust you and engage in interactions with you, as they are confident that those things are less likely to happen to them. second, if you personally are being coerced or pressured by another entity, then saying "i have no power to do this even if i wanted to" is an important bargaining chip, as it discourages that entity from trying to compel you to do it. a major category of pressure or coercion that application developers are at risk of is that by governments, so "censorship resistance" ties strongly into this kind of argument. public blockchains are open, and therefore are likely to be used by very many entities and gain some network effects. to give a particular example, consider the case of domain name escrow. currently, if a wants to sell a domain to b, there is the standard counterparty risk problem that needs to be resolved: if a sends first, b may not send the money, and if b sends first then a might not send the domain. to solve this problem, we have centralized escrow intermediaries, but these charge fees of three to six percent. however, if we have a domain name system on a blockchain, and a currency on the same blockchain, then we can cut costs to near-zero with a smart contract: a can send the domain to a program which immediately sends it to the first person to send the program money, and the program is trusted because it runs on a public blockchain. note that in order for this to work efficiently, two completely heterogeneous asset classes from completely different industries must be on the same database not a situation which can easily happen with private ledgers. another similar example in this category is land registries and title insurance, although it is important to note that another route to interoperability is to have a private chain that the public chain can verify, btcrelay-style, and perform transactions cross-chain. in some cases, these advantages are unneeded, but in others they are quite powerful powerful enough to be worth 3x longer confirmation times and paying 0.03foratransaction(or,oncescalabilitytechnologycomesintoplay,0.03 for a transaction (or, once scalability technology comes into play, 0.03foratransaction(or,oncescalabilitytechnologycomesintoplay,0.0003 for a transaction). note that by creating privately administered smart contracts on public blockchains, or cross-chain exchange layers between public and private blockchains, one can achieve many kinds of hybrid combinations of these properties. the solution that is optimal for a particular industry depends very heavily on what your exact industry is. in some cases, public is clearly better; in others, some degree of private control is simply necessary. as is often the case in the real world, it depends. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements more uncle statistics | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search more uncle statistics posted by vitalik buterin on september 25, 2015 research & development the following are some interesting results on the performance of different miners over the course of the first 280,000 blocks of the ethereum blockchain. for this timespan i have collected the list of block and uncle coinbase addresses; raw data can be found here for blocks and here for uncles, and from this we can glean a lot of interesting information particularly about stale rates and how well-connected the different miners and pools are. first off, the scatter plot: what we clearly see here are a few primary trends. first of all, uncle rates are quite low compared to olympic; altogether we have seen 20750 uncles with 280000 blocks, or an uncle rate of 7.41% (if you compute this inclusively, ie. uncles as a percentage of all blocks rather than uncles per block, you get 6.89%) in short, not that much higher than similar figures for bitcoin even back in 2011, when its mining ecosystem was more similar to ethereum's with cpu and gpus still being dominant and with a low transaction volume. note that this does not mean that miners are getting only 93.11% of the revenue that they would be if they were infinitely well-connected to everyone else; ethereum's uncle mechanic effectively cuts out ~87% of the difference, so the actual "average loss" from bad connectivity is only ~0.9%. that said, these losses will increase for two reasons once the network starts seeing more transactions: first, the uncle mechanic works with base block rewards only, not transaction fees, and second, larger blocks necessarily lead to longer propagation times. second, we can see that there is a general trend that larger miners have lower uncle rates. this is, of course, to be expected, though it is important to dissect (1) why this happens, and (2) to what extent this is actually a real effect and not simply a statistical artefact of the fact that smaller samples tend to have more extreme results. segregating by miner size, the statistics are as follows: number of blocks mined average uncle rate <= 10 0.127 10-100 0.097 100-1000 0.087 1000-10000 0.089* >= 10000 0.055 * this result is arguably heavily skewed by a single outlier, the likely broken miner that is the dot on the chart at 4005 blocks mined, 0.378 uncle rate; not including this miner we get an average uncle rate of 0.071 which seems much more in line with the general trend. there are four primary hypotheses that can explain these results: professionalism disparity: large miners are professional operations and have more resources available to invest in improving their overall connectivity to the network (eg. by purchasing better wireless, by watching more carefully to see if their uncle rates are highly suboptimal due to networking issues), and thus have higher efficiency. small miners on the other hand tend to be hobbyists on their laptops, and may not be particularly well-connected to the internet. last-block effect: the miner that produced the last block "finds out" about the block immediately rather than after waiting ~1 second for it to propagate through the network, and thus gains an advantage in finding the next block pool efficiency: the very large miners are pools, and pools are for some reason likely related to networking more efficient than solo miners. time period differences: pools and other very large miners were not active on the first day of the blockchain, when block times were very fast and uncle rates were very high. the last-block effect clearly does not explain the entire story. if it was 100% of the cause, then we would actually see a linear decrease in efficiency: miners that mined 1 block might see an 8% uncle rate, miners that mined 28000 (ie. 10% of all) blocks would see a 7.2% uncle rate, miners that mined 56000 blocks would see a 6.4% uncle rate, etc; this is because miners that mined 20% of the blocks would have mined the latest block 20% of the time, and thus benefit from a 0% expected uncle rate 20% of the time hence the 20% reduction from 8% to 6.4%. the difference between miners that mined 1 block and miners that mined 100 blocks would be negligible. in reality, of course, the decrease in stale rates with increasing size seems to be almost perfectly logarithmic, a curve that seems much more consistent with a professionalism disparity theory than anything else. the time period difference theory is also supported by the curve, though it's important to note that only ~1600 uncles (ie. 8% of all uncles and 0.6% of all blocks) were mined during those first hectic two days when uncle rates were high and so that can at most account for ~0.6% of the uncle rates altogether. the fact that professionalism disparity seems to dominate is in some sense an encouraging sign, especially since (i) the factor matters more at small to medium scales than it does at medium to large scales, and (ii) individual miners tend to have countervailing economic factors that outweigh their reduced efficiency particularly, the fact that they are using hardware that they largely already paid for. now, what about the jump from 7.1% at 1000-10000 blocks to 5.5% for everyone above that? the last-block effect can account for about 40% of the effect, but not all of it (quick math: the average miner in the former cohort has a network share of 1%, in the latter cohort 10%, and the difference of 9% should project a decrease from 7.1* to 7.1% * 0.93 = 6.4%), though given the small number of miners it's important to note that any finding here should be taken as being highly tentative at best. the key characteristic of the miners above 10000 blocks, quite naturally, is that they are pools (or at least three of the five; the other two are solo miners though they are the smallest ones). interestingly enough, the two non-pools have uncle rates of 8.1% and 3.5% respectively, a weighted average of 6.0% which is not much different from the 5.4% weighted average stale rate of the three pools; hence, in general, it seems as though the pools are very slightly more efficient than the solo miners, but once again the finding should not be taken as statistically significant; even though the sample size within each pool is very large, the sample size of pools is small. what's more, the more efficient mining pool is not actually the largest one (nanopool) it's suprnova. this leads us to an interesting question: where do the efficiencies and inefficiencies of pooled mining come from? on one hand, pools are likely very well connected to the network and do a good job of spreading their own blocks; they also benefit from a weaker version of the last-block effect (weaker version because there is still the single-hop round trip from miner to pool to miner). on the other hand, the delay in getting work from a pool after creating a block should slightly increase one's stale rate: assuming a network latency of 200ms, by about 1%. it's likely that these forces roughly cancel out. the third key thing to measure is: just how much of the disparities that we see is because of a genuine inequality in how well-connected miners are, and how much is random chance? to check this, we can do a simple statistical test. here are the deciles of the uncle rates of all miners that produced more than 100 blocks (ie. the first number is the lowest uncle rate, the second number is the 10th percentile, the third is the 20th percentile and so on until the last number is the highest): [0.01125703564727955, 0.03481012658227848, 0.04812518452908179, 0.0582010582010582, 0.06701030927835051, 0.07642487046632124, 0.0847457627118644, 0.09588299024918744, 0.11538461538461539, 0.14803625377643503, 0.3787765293383271] here are the deciles generated by a random model where every miner has a 7.41% "natural" stale rate and all disparities are due to some being lucky or unlucky: [0.03, 0.052980132450331126, 0.06140350877192982, 0.06594885598923284, 0.06948640483383686, 0.07207207207207207, 0.07488986784140969, 0.078125, 0.08302752293577982, 0.09230769230769231, 0.12857142857142856] so we get roughly half of the effect. the other half actually does come from genuine connectivity differences; particularly, if you do a simple model where "natural" stale rates are random variables with a normal distribution around a mean of 0.09, standard deviation 0.06 and hard minimum 0 you get: [0, 0.025374105400130124, 0.05084745762711865, 0.06557377049180328, 0.07669616519174041, 0.09032875837855091, 0.10062893081761007, 0.11311861743912019, 0.13307984790874525, 0.16252390057361377, 0.21085858585858586] this is pretty close, although is does grow too fast on the low side and slowly on the high side; in reality, it seems that the best-fit "natural stale rate distribution" exhibits positive skewness, which we would expect given the dimishing returns in spending increasing effort on making oneself more and more well-connected to the network. all in all, the effects are not very large; especially when divided by 8 after the uncle mechanism is taken into account, the disparities are much smaller than the disparities in electricity costs. hence, the best approaches to improving decentralization moving forward are arguably highly concentrated in coming up with more decentralized alternatives to mining pools; perhaps mining pools implementing something like meni rosenfeld's multi-pps may be a medium term solution. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements olympic: frontier pre-release | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search olympic: frontier pre-release posted by vitalik buterin on may 9, 2015 protocol announcements what began all the way back on midnight of february 1st 2014, is coming to a close: we are happy to announce the release of the ninth and last in the ethereum proof-of-concept series. we invite the community to participate in the ongoing proof-of-concept ix testnet in our present release, olympic, made available now. the purpose of olympic is to reward people who try to test the limits of the ethereum blockchain during the pre-release period, spamming the network with transactions and doing crazy things with the state, so that we can see how the network holds up under high levels of load. at the same time, application developers, data providers, exchanges, and users are encouraged to develop and deploy on the testnet and run nodes and if you have multiple virtual private servers, spin up as many nodes as you can. olympic will feature a total prize fund of up to 25,000 ether. there will be four categories of prize as well as a grand prize for the first person to create a substantial fork on the testnet. the four categories of prize will be transaction activity, virtual machine usage, mining prowess and general punishment. each category will include a main prize of 2,500 ether, as well as one or more smaller prizes at 100 1,000 ether and possibly tiny rewards of 0.1-5 ether simply for participating. each of the categories will be judged by vitalik, gavin and jeff, likely with substantial help from automated blockchain analysis tools. in addition to the ether, like the bounty programme, all prize-winners will be entitled to have their name immortalised in the ethereum genesis block. please note that to be considered for a prize, send an e-mail describing a claim to olympic@ethereum.org. this is also the last phase of the ethereum development process before the frontier release, as the network is currently proving to be highly stable at its current size of 20-100 nodes, all major clients have been staying in consensus and we are approaching code freeze pending testing and auditing inputs. it is expected to last close to 14 days, though we reserve the right to shorten or lengthen it based on technical considerations. when we deem that we are ready, we will provide a 48 hour countdown for the frontier 1.0 launch. binaries and source are available here: https://github.com/ethereum/go-ethereum/releases/tag/v0.9.18 a work-in-progress guide to frontier is available here: http://ethereum.gitbooks.io/frontier-guide/ transaction activity this includes activity related to sending and receiving transactions. examples of things we might reward include: the account that sends the highest total number of transactions the account that contributes the highest total of gas usage the account that receives the highest total number of transactions the account that contributes the highest bloat (as measured in bytes) to the blockchain in terms of transaction trie contents the account with the lowest address (in terms of lexicographic order or numerical representation; the two orders are equivalent) that sends at least one transaction the two accounts that send at least one transaction whose addresses are closest to each other virtual machine usage this includes activity relating to using the virtual machine. examples of things we might reward include: the account that makes the highest number of calls of any opcode the account that makes the highest number of calls of any opcode within a single transaction the account that sends the single transaction that takes the longest for a particular client to execute the account that sends the single transaction with the highest ratio of time to execute per unit gas consumed the account that receives the highest total number of messages the account that reverts the highest total amount of gas usage the account that creates the contract with the lowest address (in terms of lexicographic order or numerical representation; the two orders are equivalent) the account that sends the transaction with the largest (as measured in bytes) encoded receipt rlp the account that produces the largest number of transactions whose bloom is a (distinct) prime number a creator of a contract that contains, for example, a sudoku solver, taking an array of 81 values as input and outputting the array of 81 values corresponding to a solved sudoku mining prowess this includes activity relating to mining; miners will be identified by coinbase address. examples of things we might reward include: the miner that produces the largest block in the main chain (as measured in bytes) the miner that contributes the largest total number of bytes of block data to the main chain the miner that mines the largest number of blocks in a row the miner that includes the most transactions the miner that produces the block that takes the longest amount of time for a particular client to process general punishment this includes mucking around with the state in crazy ways, getting contracts to do crazy things. examples of things we might reward include: the sender/miner of the transaction/block that adds the largest number of items to contract storage the creator of the contract with the largest code (externally-owned accounts only) the creator of the contract with the most filled items in storage the creator of the contract that contains a key/value pair with the highest merkle tree depth (ie. the largest number of hash lookups before getting to the value) the creator of the contract that suicides while containing the most filled items in storage the creator of the contract that makes the largest number of consecutive storage updates such that the storage root at each step is a prime number. note that the above categories are by no means an exhaustive list of things that are eligible for prizes. if you do anything outside of the above that you feel merits consideration, please email us. the grand prize a grand prize of at least 5,000 ether will be shared between the miner(s) who manage to create a substantial fork between the go and c++ clients. the fork must be on the main chain; one client must accept the block with the other client rejecting it. smaller prizes for forks between go/c++ and python may also be available. good luck, and we look forward to seeing and hearing what you all come up with! the ethereum core dev team previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements understanding serenity, part 2: casper | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search understanding serenity, part 2: casper posted by vitalik buterin on december 28, 2015 research & development special thanks to vlad zamfir for introducing the idea of by-block consensus and convincing me of its merits, alongside many of the other core ideas of casper, and to vlad zamfir and greg meredith for their continued work on the protocol in the last post in this series, we discussed one of the two flagship feature sets of serenity: a heightened degree of abstraction that greatly increases the flexibility of the platform and takes a large step in moving ethereum from "bitcoin plus turing-complete" to "general-purpose decentralized computation". now, let us turn our attention to the other flagship feature, and the one for which the serenity milestone was originally created: the casper proof of stake algorithm. consensus by bet the keystone mechanism of casper is the introduction of a fundamentally new philosophy in the field of public economic consensus: the concept of consensus-by-bet. the core idea of consensus-by-bet is simple: the protocol offers opportunities for validators to bet against the protocol on which blocks are going to be finalized. a bet on some block x in this context is a transaction which, by protocol rules, gives the validator a reward of y coins (which are simply printed to give to the validator out of thin air, hence "against the protocol") in all universes in which block x was processed but which gives the validator a penalty of z coins (which are destroyed) in all universes in which block x was not processed. the validator will wish to make such a bet only if they believe block x is likely enough to be processed in the universe that people care about that the tradeoff is worth it. and then, here's the economically recursive fun part: the universe that people care about, ie. the state that users' clients show when users want to know their account balance, the status of their contracts, etc, is itself derived by looking at which blocks people bet on the most. hence, each validator's incentive is to bet in the way that they expect others to bet in the future, driving the process toward convergence. a helpful analogy here is to look at proof of work consensus a protocol which seems highly unique when viewed by itself, but which can in fact be perfectly modeled as a very specific subset of consensus-by-bet. the argument is as follows. when you are mining on top of a block, you are expending electricity costs e per second in exchange for receiving a chance p per second of generating a block and receiving r coins in all forks containing your block, and zero rewards in all other chains: hence, every second, you receive an expected gain of p*r-e on the chain you are mining on, and take a loss of e on all other chains; this can be interpreted as taking a bet at e:p*r-e odds that the chain you are mining on will "win"; for example, if p is 1 in 1 million, r is 25 btc ~= $10000 usd and e is $0.007, then your gains per second on the winning chain are 0.000001 * 10000 0.007 = 0.003, your losses on the losing chain are the electricity cost of 0.007, and so you are betting at 7:3 odds (or 70% probability) that the chain you are mining on will win. note that proof of work satisfies the requirement of being economically "recursive" in the way described above: users' clients will calculate their balances by processing the chain that has the most proof of work (ie. bets) behind it. consensus-by-bet can be seen as a framework that encompasses this way of looking at proof of work, and yet also can be adapted to provide an economic game to incentivize convergence for many other classes of consensus protocols. traditional byzantine-fault-tolerant consensus protocols, for example, tend to have a concept of "pre-votes" and "pre-commits" before the final "commit" to a particular result; in a consensus-by-bet model, one can make each stage be a bet, so that participants in the later stages will have greater assurance that participants in the earlier stages "really mean it". it can also be used to incentivize correct behavior in out-of-band human consensus, if that is needed to overcome extreme circumstances such as a 51% attack. if someone buys up half the coins on a proof-of-stake chains, and attacks it, then the community simply needs to coordinate on a patch where clients ignore the attacker's fork, and the attacker and anyone who plays along with the attacker automatically loses all of their coins. a very ambitious goal would be to generate these forking decisions automatically by online nodes if done successfully, this would also subsume into the consensus-by-bet framework the underappreciated but important result from traditional fault tolerance research that, under strong synchrony assumptions, even if almost all nodes are trying to attack the system the remaining nodes can still come to consensus. in the context of consensus-by-bet, different consensus protocols differ in only one way: who is allowed to bet, at what odds and how much? in proof of work, there is only one kind of bet offered: the ability to bet on the chain containing one's own block at odds e:p*r-e. in generalized consensus-by-bet, we can use a mechanism known as a scoring rule to essentially offer an infinite number of betting opportunities: one infinitesimally small bet at 1:1, one infinitesimally small bet at 1.000001:1, one infinitesimally small bet at 1.000002:1, and so forth. a scoring rule as an infinite number of bets. one can still decide exactly how large these infinitesimal marginal bets are at each probability level, but in general this technique allows us to elicit a very precise reading of the probability with which some validator thinks some block is likely to be confirmed; if a validator thinks that a block will be confirmed with probability 90%, then they will accept all of the bets below 9:1 odds and none of the bets above 9:1 odds, and seeing this the protocol will be able to infer this "opinion" that the chance the block will be confirmed is 90% with exactness. in fact, the revelation principle tells us that we may as well ask the validators to supply a signed message containing their "opinion" on the probability that the block will be confirmed directly, and let the protocol calculate the bets on the validator's behalf. thanks to the wonders of calculus, we can actually come up with fairly simple functions to compute a total reward and penalty at each probability level that are mathematically equivalent to summing an infinite set of bets at all probability levels below the validator's stated confidence. a fairly simple example is s(p) = p/(1-p) and f(p) = (p/(1-p))^2/2 where s computes your reward if the event you are betting on takes place and f computes your penalty if it does not. a key advantage of the generalized approach to consensus-by-bet is this. in proof of work, the amount of "economic weight" behind a given block increases only linearly with time: if a block has six confirmations, then reverting it only costs miners (in equilibrium) roughly six times the block reward, and if a block has six hundred confirmations then reverting it costs six hundred times the block reward. in generalized consensus-by-bet, the amount of economic weight that validators throw behind a block could increase exponentially: if most of the other validators are willing to bet at 10:1, you might be comfortable sticking your neck out at 20:1, and once almost everyone bets 20:1 you might go for 40:1 or even higher. hence, a block may well reach a level of "de-facto complete finality", where validators' entire deposits are at stake backing that block, in as little as a few minutes, depending on how brave the validators are (and how much the protocol incentivizes them to be). 50000-foot view summary: the blockchain is a prediction market on itself. blocks, chains and consensus as tug of war another unique component of the way that casper does things is that rather than consensus being by-chain as is the case with current proof of work protocols, consensus is by-block: the consensus process comes to a decision on the status of the block at each height independently of every other height. this mechanism does introduce some inefficiencies particularly, a bet must register the validator's opinion on the block at every height rather than just the head of the chain but it proves to be much simpler to implement strategies for consensus-by-bet in this model, and it also has the advantage that it is much more friendly to high blockchain speed: theoretically, one can even have a block time that is faster than network propagation with this model, as blocks can be produced independently of each other, though with the obvious proviso that block finalization will still take a while longer. in by-chain consensus, one can view the consensus process as being a kind of tug-of-war between negative infinity and positive infinity at each fork, where the "status" at the fork represents the number of blocks in the longest chain on the right side minus the number of blocks on the left side: clients trying to determine the "correct chain" simply move forward starting from the genesis block, and at each fork go left if the status is negative and right if the status is positive. the economic incentives here are also clear: once the status goes positive, there is a strong economic pressure for it to converge to positive infinity, albeit very slowly. if the status goes negative, there is a strong economic pressure for it to converge to negative infinity. incidentally, note that under this framework the core idea behind the ghost scoring rule becomes a natural generalization instead of just counting the length of the longest chain toward the status, count every block on each side of the fork: in by-block consensus, there is once again the tug of war, though this time the "status" is simply an arbitrary number that can be increased or decreased by certain actions connected to the protocol; at every block height, clients process the block if the status is positive and do not process the block if the status is negative. note that even though proof of work is currently by-chain, it doesn't have to be: one can easily imagine a protocol where instead of providing a parent block, a block with a valid proof of work solution must provide a +1 or -1 vote on every block height in its history; +1 votes would be rewarded only if the block that was voted on does get processed, and -1 votes would be rewarded only if the block that was voted on does not get processed: of course, in proof of work such a design would not work well for one simple reason: if you have to vote on absolutely every previous height, then the amount of voting that needs to be done will increase quadratically with time and fairly quickly grind the system to a halt. with consensus-by-bet, however, because the tug of war can converge to complete finality exponentially, the voting overhead is much more tolerable. one counterintuitive consequence of this mechanism is the fact that a block can remain unconfirmed even when blocks after that block are completely finalized. this may seem like a large hit in efficiency, as if there is one block whose status is flip-flopping with ten blocks on top of it then each flip would entail recalculating state transitions for an entire ten blocks, but note that in a by-chain model the exact same thing can happen between chains as well, and the by-block version actually provides users with more information: if their transaction was confirmed and finalized in block 20101, and they know that regardless of the contents of block 20100 that transaction will have a certain result, then the result that they care about is finalized even though parts of the history before the result are not. by-chain consensus algorithms can never provide this property. so how does casper work anyway? in any security-deposit-based proof of stake protocol, there is a current set of bonded validators, which is kept track of as part of the state; in order to make a bet or take one of a number of critical actions in the protocol, you must be in the set so that you can be punished if you misbehave. joining the set of bonded validators and leaving the set of bonded validators are both special transaction types, and critical actions in the protocol such as bets are also transaction types; bets may be transmitted as independent objects through the network, but they can also be included into blocks. in keeping with serenity's spirit of abstraction, all of this is implemented via a casper contract, which has functions for making bets, joining, withdrawing, and accessing consensus information, and so one can submit bets and take other actions simply by calling the casper contract with the desired data. the state of the casper contract looks as follows: the contract keeps track of the current set of validators, and for each validator it keeps track of six primary things: the return address for the validator's deposit the current size of the validator's deposit (note that the bets that the validator makes will increase or decrease this value) the validator's validation code the sequence number of the most recent bet the hash of the most recent bet the validator's opinion table the concept of "validation code" is another abstraction feature in serenity; whereas other proof of stake protocols require validators to use one specific signature verification algorithm, the casper implementation in serenity allows validators to specify a piece of code that accepts a hash and a signature and returns 0 or 1, and before accepting a bet checks the hash of the bet against its signature. the default validation code is an ecdsa verifier, but one can also experiment with other verifiers: multisig, threshold signatures (potentially useful for creating decentralized stake pools!), lamport signatures, etc. every bet must contain a sequence number one higher than the previous bet, and every bet must contain a hash of the previous bet; hence, one can view the series of bets made by a validator as being a kind of "private blockchain"; viewed in that context, the validator's opinion is essentially the state of that chain. an opinion is a table that describes: what the validator thinks the most likely state root is at any given block height what the validator thinks the most likely block hash is at any given block height (or zero if no block hash is present) how likely the block with that hash is to be finalized a bet is an object that looks like this: the key information is the following: the sequence number of the bet the hash of the previous bet a signature a list of updates to the opinion the function in the casper contract that processes a bet has three parts to it. first, it validates the sequence number, previous hash and signature of a bet. next, it updates the opinion table with any new information supplied by the bet. a bet should generally update a few very recent probabilities, block hashes and state roots, so most of the table will generally be unchanged. finally, it applies the scoring rule to the opinion: if the opinion says that you believe that a given block has a 99% chance of finalization, and if, in the particular universe that this particular contract is running in, the block was finalized, then you might get 99 points; otherwise you might lose 4900 points. note that, because the process of running this function inside the casper contract takes place as part of the state transition function, this process is fully aware of what every previous block and state root is at least within the context of its own universe; even if, from the point of view of the outside world, the validators proposing and voting on block 20125 have no idea whether or not block 20123 will be finalized, when the validators come around to processing that block they will be or, perhaps, they might process both universes and only later decide to stick with one. in order to prevent validators from providing different bets to different universes, we have a simple slashing condition: if you make two bets with the same sequence number, or even if you make a bet that you cannot get the casper contract to process, you lose your entire deposit. withdrawing from the validator pool takes two steps. first, one must submit a bet whose maximum height is -1; this automatically ends the chain of bets and starts a four-month countdown timer (20 blocks / 100 seconds on the testnet) before the bettor can recover their funds by calling a third method, withdraw. withdrawing can be done by anyone, and sends funds back to the same address that sent the original join transaction. block proposition a block contains (i) a number representing the block height, (ii) the proposer address, (iii) a transaction root hash and (iv) a signature. for a block to be valid, the proposer address must be the same as the validator that is scheduled to generate a block for the given height, and the signature must validate when run against the validator's own validation code. the time to submit a block at height n is determined by t = g + n * 5 where g is the genesis timestamp; hence, a block should ordinarily appear every five seconds. an nxt-style random number generator is used to determine who can generate a block at each height; essentially, this involves taking missing block proposers as a source of entropy. the reasoning behind this is that even though this entropy is manipulable, manipulation comes at a high cost: one must sacrifice one's right to create a block and collect transaction fees in order to manipulate it. if it is deemed absolutely necessary, the cost of manipulation can be increased several orders of magnitude further by replacing the nxt-style rng with a randao-like protocol. the validator strategy so how does a validator operate under the casper protocol? validators have two primary categories of activity: making blocks and making bets. making blocks is a process that takes place independently from everything else: validators gather transactions, and when it comes time for them to make a block, they produce one, sign it and send it out to the network. the process for making bets is more complicated. the current default validator strategy in casper is one that is designed to mimic aspects of traditional byzantine-fault-tolerant consensus: look at how other validators are betting, take the 33rd percentile, and move a step toward 0 or 1 from there. to accomplish this, each validator collects and tries to stay as up-to-date as possible on the bets being made by all other validators, and keeps track of the current opinion of each one. if there are no or few opinions on a particular block height from other validators, then it follows an initial algorithm that looks roughly as follows: if the block is not yet present, but the current time is still very close to the time that the block should have been published, bet 0.5 if the block is not yet present, but a long time has already passed since the block should have been published, bet 0.3 if the block is present, and it arrived on time, bet 0.7 if the block is present, but it arrived either far too early or far too late, bet 0.3 some randomness is added in order to help prevent "stuck" scenarios, but the basic principle remains the same. if there are already many opinions on a particular block height from other validators, then we take the following strategy: let l be the value such that two thirds of validators are betting higher than l. let m be the median (ie. the value such that half of validators are betting higher than m). let h be the value such that two thirds of validators are betting lower than h. let e(x) be a function that makes x more "extreme", ie. pushes the value away from 0.5 and toward 1. a simple example is the piecewise function e(x) = 0.5 + x / 2 if x > 0.5 else x / 2. if l > 0.8, bet e(l) if h < 0.2, bet e(h) otherwise, bet e(m), though limit the result to be within the range [0.15, 0.85] so that less than 67% of validators can't force another validator to move their bets too far validators are free to choose their own level of risk aversion within the context of this strategy by choosing the shape of e. a function where f(e) = 0.99999 for e > 0.8 could work (and would in fact likely provide the same behavior as tendermint) but it creates somewhat higher risks and allows hostile validators making up a large portion of the bonded validator set to trick these validators into losing their entire deposit at a low cost (the attack strategy would be to bet 0.9, trick the other validators into betting 0.99999, and then jump back to betting 0.1 and force the system to converge to zero). on the other hand, a function that converges very slowly will incur higher inefficiencies when the system is not under attack, as finality will come more slowly and validators will need to keep betting on each height longer. now, how does a client determine what the current state is? essentially, the process is as follows. it starts off by downloading all blocks and all bets. it then uses the same algorithm as above to construct its own opinion, but it does not publish it. instead, it simply looks at each height sequentially, processing a block if its probability is greater than 0.5 and skipping it otherwise; the state after processing all of these blocks is shown as the "current state" of the blockchain. the client can also provide a subjective notion of "finality": when the opinion at every height up to some k is either above 99.999% or below 0.001%, then the client considers the first k blocks finalized. further research there is still quite a bit of research to do for casper and generalized consensus-by-bet. particular points include: coming up with results to show that the system economically incentivizes convergence, even in the presence of some quantity of byzantine validators determining optimal validator strategies making sure that the mechanism for including the bets in blocks is not exploitable increasing efficiency. currently, the poc1 simulation can handle ~16 validators running at the same time (up from ~13 a week ago), though ideally we should push this up as much as possible (note that the number of validators the system can handle on a live network should be roughly the square of the performance of the poc, as the poc runs all nodes on the same machine). the next article in this series will deal with efforts to add a scaffolding for scalability into serenity, and will likely be released around the same time as poc2. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements how do you know ethereum is secure? | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search how do you know ethereum is secure? posted by jutta steiner on july 7, 2015 research & development as i'm writing this, i’m sitting in the london office and pondering how to give you a good overview about the work we’ve been doing to secure ethereum’s protocols, clients and p2p-network. as you might remember, i joined the ethereum team at the end of last year to manage the security audit. as spring has passed and summer arrived and meanwhile several audits finished, it’s now a good time for me to share some results from the inspection of the world computer’s machine room. ;-) this much is clear, as much as the delivery of the clients is an elaborate product development process, it is an exciting yet heavily complex research effort. the latter is the reason why even the best planned development schedule is subject to change as we discover more about our problem domain. the security audit started at the end of last year with the development of a general strategy for ensuring maximum security for ethereum. as you know, we have a security driven, rather than a schedule driven development process. with this in mind, we put together a multi-tiered audit approach consisting of: analyses of the new protocols and algorithms by established blockchain researchers and specialised software security companies end-to-end audit of protocols and implementation by a world-class expert security consultancy (go followed by c++ and a basic audit for the educational python client), as well as the bug bounty program. the analyses of the new protocols and algorithms covered topics like the security of: the gas economics the newly devised asic-resistant proof of work puzzle as well as the economic incentivisation of mining nodes. the “crowd-sourced” audit component started around christmas along with our bug bounty program. we had set aside an 11-digit satoshi amount to reward people who found bugs in our code. we’ve seen very high quality submissions to our bug bounty program and hunters received corresponding rewards. the bug bounty program is is still running and we need further submissions to use up the allocated budget... the first major security audit (covering the gas economics and pow puzzle) by security consultancy least authority was started in january and continued until the end of winter. we are very glad that we agreed with most of our external auditors that those audit reports will be publicly available once the audit work and fixing of the findings is completed. so along with this blog post, we are delighted to present the least authority audit report and accompanying blog post.  in addition, the report contains helpful recommendations for ðapp developers to ensure secure design and deployment of contracts. we expect to publish further reports as they become available. we have also engaged another software security firm at the beginning of the year to provide audit coverage on the go implementation. given the increased security that comes with multiple clients and as gav mentioned in his previous post, we have also decided to give the python and c++ audit a lightweight security audit starting early july. the c++ code will receive a full audit right after – our goal with this approach is to ensure several available audited clients as early as possible during the release process. we kicked off this most encompassing audit for the go client, aka the “end to end audit”, in february with a one-week workshop that would be followed by weeks of regular check-in calls and weekly audit reports. the audit was embedded in a comprehensive process for bug tracking and fixing, managed and thoroughly tracked on github by gustav with christoph and dimitry coding up the corresponding required tests. as the name implies, the end-to-end audit was scoped to cover “everything” (from networking to the ethereum vm to syncing layer to pow) so that at least one auditor would have cross checked the various core layers of ethereum. one of the consultants recently summarized the situation pretty succinctly: “to be honest, the testing needs of ethereum are more complex than anything i’ve looked at before”. as gav reported in his last blog post, because of the significant changes in the networking and syncing strategy we eventually decided to commission further audit work for go – which we are about to finish this week. the kick-off for the end-to-end c++ and basic python audits is taking place now. the audit work with subsequent bug fixing and regression testing as well as related refactoring and redesign (of networking and syncing layer) make up the majority of work that’s keeping the developers busy right now. likewise, fixing of findings, redesign and regression testing are the reason for the delay in the delivery. in addition, the olympic testing phase has taught us a great deal about resiliency under various scenarios, such as slow connections, bad peers, odd behaving peers and outdated peers. the greatest challenge so far has been fighting off and recovering from forks. we learnt a lot from the recovery attempts in terms of required processes when it comes to dealing with these type of scenarios and incidents. it might not come as a surprise that the various audits represent a significant expenditure – and we think money that could not be better invested. as we draw closer to release, security and reliability is increasingly uppermost in our minds, particularly given the handful of critical issues found in the olympic test release. we are very grateful for the enthusiasm and thorough work that all auditors have done so far. their work helped us sharpen the specification in the yellow paper and to weed out ambiguity and fix several subtle issues, and they helped with identifying a number of implementation bugs. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements on slow and fast block times | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on slow and fast block times posted by vitalik buterin on september 14, 2015 research & development one of the largest sources of confusion in the question of blockchain security is the precise effect of the block time. if one blockchain has a block time of 10 minutes, and the other has an estimated block time of 17 seconds, then what exactly does that mean? what is the equivalent of six confirmations on the 10-minute blockchain on the 17-second blockchain? is blockchain security simply a matter of time, is it a matter of blocks, or a combination of both? what security properties do more complex schemes have? note: this article will not go into depth on the centralization risks associated with fast block times; centralization risks are a major concern, and are the primary reason not to push block times all the way down to 1 second despite the benefits, and are discussed at much more length in this previous article; the purpose of this article is to explain why fast block times are desirable at all. the answer in fact depends crucially on the security model that we are using; that is, what are the properties of the attackers that we are assuming exist? are they rational, byzantine, economically bounded, computationally bounded, able to bribe ordinary users or not? in general, blockchain security analysis uses one of three different security models: normal-case model: there are no attackers. either everyone is altruistic, or everyone is rational but acts in an uncoordinated way. byzantine fault tolerance model: a certain percentage of all miners are attackers, and the rest are honest altruistic people. economic model: there is an attacker with a budget of $x which the attacker can spend to either purchase their own hardware or bribe other users, who are rational. reality is a mix between the three; however, we can glean many insights by examining the three models separately and seeing what happens in each one. the normal case let us first start off by looking at the normal case. here, there are no attackers, and all miners simply want to happily sing together and get along while they continue progressively extending the blockchain. now, the question we want to answer is this: suppose that someone sent a transaction, and k seconds have elapsed. then, this person sends a double-spend transaction trying to revert their original transaction (eg. if the original transaction sent $50000 to you, the double-spend spends the same $50000 but directs it into another account owned by the attacker). what is the probability that the original transaction, and not the double-spend, will end up in the final blockchain? note that, if all miners are genuinely nice and altruistic, they will not accept any double-spends that come after the original transaction, and so the probability should approach 100% after a few seconds, regardless of block time. one way to relax the model is to assume a small percentage of attackers; if the block time is extremely long, then the probability that a transaction will be finalized can never exceed 1-x, where x is the percentage of attackers, before a block gets created. we will cover this in the next section. another approach is to relax the altruism assumption and instead discuss uncoordinated rationality; in this case, an attacker trying to double-spend can bribe miners to include their double-spend transaction by placing a higher fee on it (this is essentially peter todd's replace-by-fee). hence, once the attacker broadcasts their double-spend, it will be accepted in any newly created block, except for blocks in chains where the original transaction was already included. we can incorporate this assumption into our question by making it slightly more complex: what is the probability that the original transaction has been placed in a block that will end up as part of the final blockchain? the first step to getting to that state is getting included in a block in the first place. the probability that this will take place after k seconds is pretty well established: unfortunately, getting into one block is not the end of the story. perhaps, when that block is created, another block is created at the same time (or, more precisely, within network latency); at that point, we can assume as a first approximation that it is a 50:50 chance which of those two blocks the next block will be built on, and that block will ultimately "win" or, perhaps, two blocks will be created once again at the same time, and the contest will repeat itself. even after two blocks have been created, it's possible that some miner has not yet seen both blocks, and that miner gets lucky and created three blocks one after the other. the possibilities are likely mathematically intractable, so we will just take the lazy shortcut and simulate them: script here the results can be understood mathematically. at 17 seconds (ie. 100% of the block time), the faster blockchain gives a probability of ~0.56: slightly smaller than the matheatically predicted 1-1/e ~= 0.632 because of the possibility of two blocks being created at the same time and one being discarded; at 600 seconds, the slower blockchain gives a probability of 0.629, only slightly smaller than the predicted 0.632 because with 10-minute blocks the probability of two blocks being created at the same time is very small. hence, we can see that faster blockchains do have a slight disadvantage because of the higher influence of network latency, but if we do a fair comparison (ie. waiting a particular number of seconds), the probability of non-reversion of the original transaction on the faster blockchain is much greater. attackers now, let's add some attackers into the picture. suppose that portion x of the network is taken up by attackers, and the remaining 1-x is made up of either altruistic or selfish but uncoordinated (barring selfish mining considerations, up to x it actually does not matter which) miners. the simplest mathematical model to use to approximate this is the weighted random walk. we start off assuming that a transaction has been confirmed for k blocks, and that the attacker, who is also a miner, now tries to start a fork of the blockchain. from there, we represent the situation with a score of k, meaning that the attacker's blockchain is k blocks behind the original chain, and at every step make the observation that there is a probability of x that the attacker will make the next block, changing the score to k-1 and a probability of 1-x that honest miners mining on the original chain will make the next block, changing the score to k+1. if we get to k = 0, that means that the original chain and the attacker's chain have the same length, and so the attacker wins. mathematically, we know that the probability of the attacker winning such a game (assuming x < 0.5 as otherwise the attacker can overwhelm the network no matter what the blockchain parameters are) is: we can combine this with a probability estimate for k (using the poisson distribution) and get the net probability of the attacker winning after a given number of seconds: script here note that for fast block times, we do have to make an adjustment because the stale rates are higher, and we do this in the above graph: we set x = 0.25 for the 600s blockchain and x = 0.28 for the 17s blockchain. hence, the faster blockchain does allow the probability of non-reversion to reach 1 much faster. one other argument that may be raised is that the reduced cost of attacking a blockchain for a short amount of time over a long amount of time means that attacks against fast blockchains may happen more frequently; however, this only slightly mitigates fast blockchains' advantage. for example, if attacks happen 10x more often, then this means that we need to be comfortable with, for example, a 99.99% probability of non-reversion, if before we were comfortable with a 99.9% probability of non-reversion. however, the probability of non-reversion approaches 1 exponentially, and so only a small number of extra confirmations (to be precise, around two to five) on the faster chain is required to bridge the gap; hence, the 17-second blockchain will likely require ten confirmations (~three minutes) to achieve a similar degree of security under this probabilistic model to six confirmations (~one hour) on the ten-minute blockchain. economically bounded attackers we can also approach the subject of attackers from the other side: the attacker has $x to spend, and can spend it on bribes, near-infinite instantaneous hashpower, or anything else. how high is the requisite x to revert a transaction after k seconds? essentially, this question is equivalent to "how much economic expenditure does it take to revert the number of blocks that will have been produced on top of a transaction after k seconds". from an expected-value point of view, the answer is simple (assuming a block reward of 1 coin per second in both cases): if we take into account stale rates, the picture actually turns slightly in favor of the longer block time: but "what is the expected economic security margin after k seconds" (using "expected" here in the formal probability-theoretic sense where it roughly means "average") is actually not the question that most people are asking. instead, the problem that concerns ordinary users is arguably one of them wanting to get "enough" security margin, and wanting to get there as quickly as possible. for example, if i am using the blockchain to purchase a $2 coffee, then a security margin of $0.03 (the current bitcoin transaction fee, which an attacker would need to outbid in a replace-by-fee model) is clearly not enough, but a security margin of $5 is clearly enough (ie. very few attacks would happen that spend $5 to steal $2 from you), and a security margin of $50000 is not much better. now, let us take this strict binary enough/not-enough model and apply it to a case where the payment is so small that one block reward on the faster blockchain is greater than the cost. the probability that we will have "enough" security margin after a given number of seconds is exactly equivalent to a chart that we already saw earlier: now, let us suppose that the desired security margin is worth between four and five times the smaller block reward; here, on the smaller chain we need to compute the probability that after k seconds at least five blocks will have been produced, which we can do via the poisson distribution: now, let us suppose that the desired security margin is worth as much as the larger block reward: here, we can see that fast blocks no longer provide an unambiguous benefit; in the short term they actually hurt your chances of getting more security, though that is compensated by better performance in the long term. however, what they do provide is more predictability; rather than a long exponential curve of possible times at which you will get enough security, with fast blocks it is pretty much certain that you will get what you need within 7 to 14 minutes. now, let us keep increasing the desired security margin further: as you can see, as the desired security margin gets very high, it no longer really matters that much. however, at those levels, you have to wait a day for the desired security margin to be achieved in any case, and that is a length of time that most blockchain users in practice do not end up waiting; hence, we can conclude that either (i) the economic model of security is not the one that is dominant, at least at the margin, or (ii) most transactions are small to medium sized, and so actually do benefit from the greater predictability of small block times. we should also mention the possibility of reverts due to unforeseen exigencies; for example, a blockchain fork. however, in these cases too, the "six confirmations" used by most sites is not enough, and waiting a day is required in order to be truly safe. the conclusion of all this is simple: faster block times are good because they provide more granularity of information. in the bft security models, this granularity ensures that the system can more quickly converge on the "correct" fork over an incorrect fork, and in an economic security model this means that the system can more quickly give notification to users of when an acceptable security margin has been reached. of course, faster block times do have their costs; stale rates are perhaps the largest, and it is of course necessary to balance the two a balance which will require ongoing research, and perhaps even novel approaches to solving centralization problems arising from networking lag. some developers may have the opinion that the user convenience provided by faster block times is not worth the risks to centralization, and the point at which this becomes a problem differs for different people, and can be pushed closer toward zero by introducing novel mechanisms. what i am hoping to disprove here is simply the claim, repeated by some, that fast block times provide no benefit whatsoever because if each block is fifty times faster then each block is fifty times less secure. appendix: eyal and sirer's bitcoin ng a recent interesting proposal presented at the scaling bitcoin conference in montreal is the idea of splitting blocks into two types: (i) infrequent (eg. 10 minute heartbeat) "key blocks" which select the "leader" that creates the next blocks that contain transactions, and (ii) frequent (eg. 10 second heartbeat) "microblocks" which contain transactions: the theory is that we can get very fast blocks without the centralization risks by essentially electing a dictator only once every (on average) ten minutes, for those ten minutes, and allowing the dictator to produce blocks very quickly. a dictator "should" produce blocks once every ten seconds, and in the case that the dictator attempts to double-spend their own blocks and create a longer new set of microblocks, a slasher-style algorithm is used where the dictator can be punished if they get caught: this is certainly an improvement over plain old ten-minute blocks. however, it is not nearly as effective as simply having regular blocks come once every ten seconds. the reasoning is simple. under the economically-bounded attacker model, it actually does offer the same probabilities of assurances as the ten-second model. under the bft model, however, it fails: if an attacker has 10% hashpower then the probability that a transaction will be final cannot exceed 90% until at least two key blocks are created. in reality, which can be modeled as a hybrid between the economic and bft scenarios, we can say that even though 10-second microblocks and 10-second real blocks have the same security margin, in the 10-second microblock case "collusion" is easier as within the 10-minute margin only one party needs to participate in the attack. one possible improvement to the algorithm may be to have microblock creators rotate during each inter-key-block phase, taking from the creators of the last 100 key blocks, but taking this approach to its logical conclusion will likely lead to reinventing full-on slasher-style proof of stake, albeit with a proof of work issuance model attached. however, the general approach of segregating leader election and transaction processing does have one major benefit: it reduces centralization risks due to slow block propagation (as key block propagation time does not depend on the size of the content-carrying block), and thus substantially increases the maximum safe transaction throughput (even beyond the margin provided through ethereum-esque uncle mechanisms), and for this reason further research on such schemes should certainly be done. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum builders: tapping into the collaborative potential | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum builders: tapping into the collaborative potential posted by mihai alisie on april 6, 2015 research & development hello, fellow ethereans. i am mihai alisie, one of the ethereum founders, and for the past year i’ve served as the vice president of the ethereum foundation and as director of ethereum switzerland. during the last few months i’ve been researching and testing a number of collaborative tools, constantly assessing the usefulness and applicability of each tool in the context of our community. why? because i think there is a lot of untapped potential in this community, and i think part of the reason that potential is untapped is the lack of a streamlined collaboration system. this article is a continuation of the proposed experiment post, expanding on a few of the core ideas while exploring the ethereum.builders collaboration system used as a project. towards the end of the article you'll also find a few project updates and a couple of suggestions on how we can re-energize things and collaborate better, as a community. hopefully some of you will find this useful or maybe even take it further, as i think that improving collaboration at the community level is an important goal worthy of time and commitment. as always, feedback, help and comments are more than welcome. the initial experiment: a collaboration catalyst "alone we can do so little, together we can do so much." —helen keller we all want to do awesome things, and we need to make it as easy to collaborate as possible so that people can focus on their ideas, rather than having to worry about being invited into a skype room just to be able to coordinate with others working on similar ideas. bellow is a reddit post that sums it up nicely: from the design stage, ethereum builders was envisioned as a tool that would lower the entry barriers, increase the community member interaction and bring the fun of working together closer to everyone. besides, with an actively engaged community, the ongoing feedback given by the group can also help the developer team test and improve features while including new people in various aspects of ethereum as a project. now, with ethereum builders you have people talking about their work, sharing knowledge, and providing peer-to-peer support, all of which also allows new developers to come on board and be up and running quickly and successfully. regarding the future ethereum sessions, i’d like to remind everyone that anyone can schedule a session. that means anyone really with a project or an interesting topic can propose a time, and if people consider the project and/or topic attractive, the rest will take care of itself. since soft-launched on march 20th, ethereum builders has now over 100 members that share knowledge and collaborate openly. awesome! the first session was a much-needed ethereum client install fest, followed by an introduction in the ethereum 101 project, a lounge session with decentral vancouver and spiking up with the awesome project groundhog presentation and brainstorming session. we also have a few cool sessions coming up, such as the “ultimate stablecoin built on ethereum” (edollar) scheduled for april 12th to blockchain based prediction markets built on ethereum (augur, date not set yet). the sessions are open for anyone, so if you find any of these topics interesting, join in or create one yourself if you’d like to propose a topic! now, in order to manage this project, we use a cocktail of collaboration tools and since it has been received well by quite a few people already, i’d like to share it with you too. if you find this useful and need some help in getting it up and running or want to say thank-you, get in touch with us. kicking it up a notch: github powered collaboration framework "talent wins games, but teamwork and intelligence win championships." —michael jordan after exploring a vast number of options, i was convinced that co-opting the existent infrastructure (i.e. github) and repurposing it (i.e. zenhub) is the best combo for our needs, taking also considerable less time to deploy as an initial experiment. in the current form, this is how things were thought: step one: supercharge github workflows here’s how we transform your usual github account into a powerful project management tool: go to https://www.zenhub.io/ or directly install the chrome extension from the webstore connect your github account with zenhub. you can check if everything went okay by checking your github application settings. you should see zenhub as one of the authorized applications. if zenhub has been correctly installed, then a new tab called "board(s)" should appear on your github repo pages. the board(s) are populated by the issues created, each new issue becoming a card on the new zenhub board tab. this is how a populated board should look: step two: enhance github interactivity with gitter besides github, we also use an application called gitter. very straightforward setup process; an incredibly well suited app for distributed communities and projects like this one. with this tool you're able to transform your favorite repos into your favorite chatrooms ^_^ get started by: signing in with your github account join the available chatrooms or create new ones. you can find listed a few ethereum chatrooms below if you want to join the fun :) step three: accelerate open source with bountysource one of the biggest problems surrounding open-source, community-based projects is resource starvation. in our case we'll bountysource to get around the resource-starvation issue and allow the community to guide itself and become self-sustainable in the process. and here’s how bountysource fits into the picture: sign in with your github account create, contribute or solve the current issues listed in the ethereum repos (and not only). the cool part about it is that once integrated, it automatically updates the github issues (title, body, label), which is great since we're using zenhub and the gitter activity panel. below you can see an automatically updated issue that had a bounty created via bountysource. and this is how it looks integrated with gitter via the activity panel on the right: you're fully set up now to chat openly with other people about ethereum (in multiple languages), contribute to all the existent issues and also create new bounties. things will get more interesting once we have the possibility of creating a dapp and/or dao, but in the meantime, i think this is a pretty cool solution as well. for those who are curious, you can see the components in action for the ξb platform as follows: zenhub board bountysource integration gitter chatroom by setting things up this way, now the github organization ethereum.builders is morphing into an aggregator of projects, offering people a way to learn, discover and join ongoing ethereum efforts. tapping into the collaborative potential “none of us is as smart as all of us.” —ken blanchard with each person getting to know other people inside this community and finding out about new projects that resonate with them, we create the favorable conditions for a vibrant open-source community and technology. for now, you can use the below as an incomplete map of the open ethereum gitter rooms: live projects/experiments ethereum builders hello (general chatroom) ethereum go/mist implementation [gh integration with the go repo] ethereum c++ implementation [no integrations] builder’s handbook [gh, zh and bs integration] ethereum 101 project [gh, zh and bs integration] ξb platform [gh, zh and bs integration] *legend: gh = github, zh = zenhub, bs = bountysource, ξb = ethereum builders currently the focus is on creating an organized source of information for people in the form of the handbook and growing the collaboration and knowledge-sharing network. the community response and analytics look encouraging and i think this shows that there are people “out there” finding this useful. interestingly enough, china seems to be our #1 readership. this comes with a big “thank you” to aurel iancu from ethereum romania, shaoping from ethchina.org and jan xie from yunbi.com for making it happen. without your help this would have not been possible. high five everyone! it's worth mentioning that the guide has also been recommended as the #2 read selection from the universe of books published through gitbook via their weekly newsletter. yay! we’re also in the process of converting ethereum.builders in a multilingual knowledge-sharing network. we started with chinese, spanish, russian, german and italian. depending on the results, we are planning to add more languages and see where it goes from there. we’re doing this in order to activate localized learning and tap into the potential existent in the meetups currently based in more than 33 countries. since the sessions are held by people from the community for the community, the focus shifts to enabling the local ethereans to make an impact in their geographical region by sharing knowledge and helping others in their mother tongue. we started by creating language categories on ethereum.builders. when creating a session, simply select the language used and then your session will be displayed under the language category. you can find below an example for sessions that are in english: it's also easy to find and navigate through the various ethereum builders multilingual gitter chatrooms. from gitter just type in "ethereum" + "language": the chatrooms are directly accessible also via the links below: eb chinese eb spanish eb russian eb german eb italian suggest one here besides this, we have also created ethereum.builders/plexus: the intention was to create a place where you can go and have the basic resources at hand to get started asap, while also having a way to interact with and ask for help from fellow ethereans in case you get stuck somewhere. next, we’ll work on making the gitbook also multi-lingual and polish it as it still feels kind of hacky at the moment. on this note, any help is welcome and if you would like to get involved in this please get in touch. we could use some assistance :) as for the future, depending on how things go, in the coming weeks we’ll shift our focus to the dapp, proto-reputation and proto-dao aspects of the experiment. if you any of these things are interesting, feel free to drop by to say hello or share any ideas/feedback/suggestions. so…what next? “the biggest room in the world is room for improvement.” —anonymous i hope to see many interesting projects making their appearance in the emergent ethereum builders mesh collaboration network. even if not perfect in the current form, this is a tool that can energize the community and serve as catalyst – but again, it’s nothing without people using it. by sharing your insights and ideas with others, besides getting exposure for your project, you’re also more likely to inspire someone to work on a project as well. and every individual is important, as anyone could possess the right piece of the puzzle and/or insight to push everything forward. here are a few proposed ways to increase activity and interaction inside the ethereum community: create an ethereum builders session for helping people install ethereum on their machines presenting the ethereum project(s) you’re working on brainstorming on ideas and projects screensharing with others while coding on ethereum syncing with other ethereum meetups running a workshop or hackathon simply talking about things in a relaxed atmosphere it's worth mentioning that even if no monetary rewards are involved in this, reputation is. we have started to keep track of the contributions and involvement in a very crude form on ethereum.builders/co-creators: we’re all excited about the upcoming launch, but in all this excitement i think it is important also to keep in mind that regardless of how great the technology is, without an energized community co-creating a multi-verse of smart contracts with it, ethereum's success is not guaranteed by any means. that being said, the game is one of the biggest (the internet) and this comes with deep ramifications in many areas of our civilization. the decentralized revolution ignited by the bittorrent protocols is reaching well outside sharing mp3s. we all saw the impact bitcoin had on the internet, in just six years, as a single blockchain-based decentralized financial application. now here comes the question: how will the internet look with thousands of running decentralized applications? i’m not sure either, but making it as easy as possible for people to actually build these decentralized applications is probably a good first step towards finding out the answer. and, we all know, third time's a charm! #web3 here we come! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ðξvcon-0 recap | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ðξvcon-0 recap posted by george hallam on december 5, 2014 events day 1 monday 24th nov ðξv: mission and processes the first day of ðξvcon-0 kicked off early at 7am with the ethereum uk communications team arriving at the venue (ethereum dev ug’s workspace in kreuzberg, berlin) to set up the 4k high quality recording equipment and arrange the space for the event.  after a quick coffee and croissant/pain au chocolat, everyone was ready for the first presentation “welcome! our mission: ðapps” which was delivered by gavin wood. gavin made it very clear within this presentation the need for decentralised applications in today’s society with two very powerful opening quotes, “progress has led to the concentration of power” and "our mission therefore is decentralisation”. following the presentation on ðapps, gavin invited everyone present to introduce themselves and explain their roles in ethereum. it was humbling to see how ethereum has attracted such a magnificent team of incredibly talented people from such a wide variety of backgrounds.  next up, stephan tual gave a talk highlighting the important role of the uk based communications team in building a strong and populous ethereum community. he also touched on the streamlining of internal communications procedures to maximise ethereum's core team efficiency. with 81 ethereum meetup/hackathons taking place from tehran to new york and with over 6000 members worldwide, the ethereum community has grown exponentially over the past 11 months. still, there is always more to be done. with the inclusion of 4 new members to the team, new educational tools on the horizon and increased interaction with the community, stephan hopes ethereum can become a worldwide phenomenon as we move forward to the release of the genesis block. straight after stephan’s talk, sven ehlert gave an insightful presentation on how scrum and agile will be utilised to again maximise the efficiency of ethereum’s software development.  this will be achieved by focusing on the key elements of the “product” that are needed for completion, and by applying incremental development goals with short deadlines for each of the individual developer teams (mist, c++, solidity etc). this will hopefully increase the overall output whilst breaking down what is a truly enormous project into manageable pieces.  finally to cap off the first day, alex van de sande gave an update on mist, the ethereum go client’s ðapp navigator. as many of you will have seen in alex’s recent mist presentation, the ðapp navigator will be an incredibly useful and powerful tool for people looking to interact with ethereum and will really help make it accessible and easy to use by everyone from the outset which is of course important for swift adoption. day 2 tuesday 25th nov languages, ðapps & architecture after monday’s introduction and team updates, day 2 would focus strongly on the direction of ethereum’s languages, ðapp creation and architecture updates. gavin wood and christian reitwießner started off with a talk on the vision of the solidity language and the roadmap to its completion as it comes closer to a language that supports all features provided by the ethereum virtual machine. you can try solidity in your browser now. this was followed by a brief chat from gavin this time focusing on the importance of secure documentation in ðapp development. the concept is to intertwine documentation and code in solidity so that the documentation can be used by the client to alert the user of any significant actions that may take place as a consequence of running the code in certain smart contracts.  marek kotewicz then gave an update on the javascript api in a workshop setting allowing a lot of to-and-fro with the audience. he explained how it works, how to use it with smart contracts and what tools will be available to enable its use and uptake in the future. after lunch, piotr zieliński presented on golem, a project that aims to use ethereum to coordinate a p2p distributed computation network for research use. participants of the network share their computing power and in return receive a token of worth, incentivising their continued participation. users can also access the resources to implement their own tasks and distribute them across the network. following piotr, sven elert took the stage again to speak further on optimising ethereum’s workflow using continuous integration. ci is a concept intended to improve the quality of whichever project it is being used with. though ethereum already uses ci, we wish to further refine the process as we move forward. to this end, sven introduced several new and different implementations to make the developers lives easier as the project moves forward. to finish off day 2, lefteris karapetsas gave an interesting presentation about using emacs for c++ ethereum development. he highlighted several plugins to use in order to improve workflow as well as the plugin he's working on for syntax highlighting on solidity for emacs. it is very cool to see personal projects increasing ethereum's team overall efficiency! day 3 wednesday 26th nov client comparisons and what is ethereum? wednesday’s presentation and panel content offered a great opportunity to get the whole team on the same page from a client perspective. with martin becze arriving from san francisco, we had representatives for each client ready for a panel talk later on in the day. the aim of panel was to increase the development team’s understanding of the different clients and highlight the minor differences between each client and how they implement the ethereum virtual machine.  the morning commenced with vinay gupta leading a workshop which had everyone present trying to come up with a definitive answer to “what is ethereum?”. it turns out it’s not as easy as we thought! each person had a chance to stand up and offer their own personal 30 second definition on what ethereum is. the answers were diverse, and it was interesting to see how people changed their angle of approach depending on which audience the explanation was aimed at.  following vinay’s workshop, martin becze brought everyone up to speed with node-ethereum a node.js project he has been working on with joseph chow in palo alto. the talk started by outlining the ethereumjs-lib and node-etheruem's architecture, then focused on areas where the client differs in structure from the other c++, go and python client implementations. jeff wilcke also gave a brief update on ethereum go client on the whiteboard before the panel discussion. the presentation space was then rearranged for the panel with gavin wood (ethereum c++), jeff wilcke (ethereum go), heiko hees (pythereum) and martin becze (node-ethereum). each took turns outlining how each client handles state transactions, moves account balances, runs the evm and saves data to accounts whilst handling sucides and out-of-gas exceptions. questions where then posed by the audience, discussions continued on late into the night as day 3 drew to a close.  day 4 thursday 27th nov robustness and security though it is exceptionally useful to have a large suite of tools available for developers and users wanting to interact with ethereum, it means nothing if the security and robustness of the network are in anyway compromised. to this end, the ethereum team is working very hard to audit the software with several external partners and ensure network stability from the release of the genesis block onwards. to this end, sven ehlert, jutta steiner and heiko hees kicked off the morning with a workshop on the stress testing project. the goal of the project is to again use continuous integration and take it one step further. with ci, the dev team can test to make sure that each client will behave correctly when it executes a contract. in this respect it will simulate how clients would interact with the blockchain in the real world. this will allow us to understand how the network will behave if the clients have different properties. for instance clients with low bandwidth and less cpu power interacting with those with more resources available. the project pays special attention to how this will affect potential forks of the blockchain. later on, the team will simulate attacks on the ethereum blockchain and network to test its durability and resilience.  after the workshop, christoph jenzsch presented on the why and how of unit testing in the ethereum project. in his talk, christoph described several different types of test that need to be carried out unit tests, integration tests and system tests. he also explained the different tools available to make it easy for every developer involved in ethereum to get working on the huge amount of testing that needs to be carried out before it’s released.  next, jutta steiner took the stage to deliver a talk on “achieving security”. she explained how our approach to security is to initiate both an internal audit and a massive external audit with established software security firms, academics, blockchain research firms and companies interested in utilising ethereum all taking part. jutta is also working on a bounty program which will be open to the community rewarding those who test and explore the protocol and software. we’ll be releasing more information on this shortly if you wish to take part. after a quick lunch, alex leverington updated us on the vision and roadmap of the libp2p multi  protocol network framework. the p2p networking protocol is geared towards security as it is entirely encrypted from end to end. he contrasted the ethereum network with the internet as we know it today, emphasising the difference between using ssl certificates and p2p anonymous networks. he also commented on ethereum’s 3 network protocols whisper, swarm and ethereum itself. all 3 protocols run on the same line, so framing/multiplexing is being used to stop them from interfering with each others bandwidth needs. a key point to take away from this is that other ðapps can potentially build their own protocols and use the multiplexing to optimise the connection as well. gavin followed up with a presentation on the what and why of whisper: the multi dht messaging system with routing privacy. consensus is expensive, and for things like pure communications there are more elegant solutions available than using blockchain tech, hence the need for whisper. to understand whisper in general, think transactions, but without the eventual archival, any necessity of being bound to what is said or automated execution & state change. he also talked about some specific use cases for whisper, such as ðapps that need to provide dark (plausible denial over perfect network traffic analysis) comms to two correspondents that know nothing of each other but a hash. this could be a ðapp for a whistleblower to communicate to a known journalist exchange some small amount of verifiable material and arrange between themselves for some other protocol (swarm, perhaps) to handle the bulk transfer. to finish off the day, daniel nagy bought us up to speed on swarm, ethereum’s decentralised file storage and transmission protocol specifically targeted toward static web content hosting. daniel also covered dht (distributed hash table kademlia). this will allow peers to connect to other peers distributed throughout the network instead of peer that are simply close to each other. by deterministically connecting to random peers, you know that each peer will have a better picture of the overall state of the network, increasing its viability. day 5 friday 28th nov blockchain theory and the future the fifth and final day of devcon 0! after a week of talking about the near future of ethereum’s development and completion, the last day would be focused on future iterations of ethereum and blockchain technology as a whole. vitalik buterin and vlad zamfir started off day five with a whiteboard presentation on ethereum 1.x. there are several problems that need to be solved as blockchain technology moves forward in the future scalability and blockchain interoperability being at the forefront of those issues. the idea of ethereum interacting with thousands of other blockchains is an attractive one, this in itself solves certain scalability problems in that work can distributed across many chains as opposed to bloating one central chain. gavin then joined vitalik and together they talked about the ethereum light-client. a light client protocol is especially important for ethereum as it will allow it to run on low capacity environments such as mobile phones, iot (internet of things) devices and browser extensions. the two explained the development process and usage of patricia merkle trees to implement it. expect the light client to be ready for the release of ethereum 1.0. after lunch, juan batiz-benet of filecoin/ipfs showed the audience the technology behind ipfs and bitswap. he also presented some ethereum/ipfs use cases such as content addressed files, ipns naming and high performance distributed filesystem and tool integration. gavin once again took the stage to reveal mix, the ethereum ide. mix is being developed in conjunction with alethzero and expected to be released in the next 12 to 18 months. it supports a rather amazing feature set, including documentation, compiler and debugger integration as you type, information on code health, valid invariant, code structure and code formatting, as well as variable values and assertion truth annotations. finally, vitalik wrapped up the event with a presentation showing some ideas for ethereum 2.0 (sharding, hypercubes) and thanked all of those who attended.  overall, ðξvcon-0 was very enriching and worthwhile for everyone who attended. as expected, the eth dev team is now  very much on the same page and ready to push forward to the release of the genesis block. all of the presentations from devcon 0 were recorded and are currently being edited together by our resident graphic designer/video editor ian meikle. starting on the 8th, a steady stream of content “the ðξvcon-0 series” will be added to the ethereum youtube channel.  previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ether sale: a statistical overview | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ether sale: a statistical overview posted by vitalik buterin on august 8, 2014 organizational the first two weeks of the ether sale are over, and we have to date received over 25000 btc from selling over 50 million eth. this marks the largest cryptographic token sale to date, and with the two endowments places eth as being the token with the 8th highest total value, even beating out the beloved dogecoin at 17.3musdvs17.3m usd vs 17.3musdvs15.5m. a total of 6670 transactions have been made, with values ranging from the minimum 0.01 btc to a high of 500 btc, and purchases continue to come in every hour. additionally, the ether sale marks the largest use of multisig to date; because of our sale, the percentage of all btc stored in multisig has shot up from 0.23% to 0.41% over the last two weeks alone in other words, the 3-of-4 private keys split between our various sites control 45% of all btc stored in multisig addresses in existence. the purpose of this post will be to provide an overview of some statistics from the sale so far. data was taken yesterday, when we had 24000 btc, and assumes that all purchases were for 2000 eth / btc (an assumption that is not strictly true, but the error term is sufficiently tiny that it can safely be discounted). first we have this spreadsheet, which shows the ether purchases over time. the individual spikes are per-block; the chart shows that the distribution is heavily divided into two clusters, with one cluster closer to the start of the sale and the other close to the end of the full-discount period. purchases drop off sharply once the new price level of 1970 eth/btc (now 1910 eth/btc) kicked in. theoretically, purchasing near the end of the full-discount period is the more optimal strategy from a naive game-theoretic model; if you purchase near the end of the full-discount period then you get the same price as people who purchased at the start, but also gain the benefit of having more information namely, a better idea of the exact percentage of all eth that you are going to get. thus, the fact that the majority of purchases occurred at the end shows that ether purchasers are generally a rather sophisticated audience which i suppose you should be if you managed to be convinced to trade your hard-earned btc for some cryptographic tokens backed by a concept of "generalized consensus computing". of course, it is important to note that there are reasons to buy at the start too. some people are participating in the sale out of a desire to support the project, and some large purchasers may have perhaps had the priming effect in mind, where putting larger sums of money (eg. bills) into a tipping jar at the very beginning increases the total amount received because it creates the impression that the recipient is significant and merits more and larger contributions. at this point, we can expect to see a declining flow that will stabilize over the next few days, and then a smaller final spike on day 42. the chart below shows the cumulative ether sold up until this point: https://docs.google.com/a/ethereum.org/spreadsheets/d/1h5w9yvp1eronp8n9uffvccz51q5dxzjaovlicaat46g/gviz/chartiframe?oid=831527247 the other interesting thing to analyze is the distribution of purchases. this spreadsheet contains a list of purchases arranged by purchase size. the largest single purchase was 500 btc (1 million ether), followed by one at 466 btc (933,580 eth) and 330 btc (660,360 eth). we have not received any requests at largepurchases@ethereum.org. if we arrange purchases by size, we get the following two graphs, one for the quantity of purchases and one for the amount of eth purchased, by purchase size: https://docs.google.com/a/ethereum.org/spreadsheets/d/1gs9pzsdmx9lk0xgskedr_aoi02riq3mpryventvum68/gviz/chartiframe?oid=168457404 https://docs.google.com/a/ethereum.org/spreadsheets/d/1gs9pzsdmx9lk0xgskedr_aoi02riq3mpryventvum68/gviz/chartiframe?oid=846945325 note that this only applies to purchases. there is also another slice of ether which will soon be distributed, which is the endowment. the portions in which the endowment is planned to be distributed are on the spreadsheet; the largest is equal to 0.922% of all ether purchased (ie. 0.369% of the total supply after five years) and the smallest is 0.004%, with 81 people total receiving a share. if you are one of the recipients, you will be contacted shortly; if you are not then there is still a second slice whose distribution has not been decided. distribution and gini indices as a final set of interesting statistics, we have calculated three gini indices: gini index of ether purchasers: 0.832207 gini index of endowment: 0.599638 gini index of entire set: 0.836251 a gini index is a common measure of inequality; the way the gini index is calculated is by drawing a chart, with both axes going from 0% to 100%, and drawing a line where the y coordinate at a particular x coordinate is calculated as the portion of all income (or wealth) which is owned by the bottom x percent of the population. the area between this curve and a diagonal line, as a portion of the area of the entire triangle under the diagonal line, is the gini index: in an ideal society of perfect equality, the coefficient would be zero; the bottom x% of the population would obviously have x% of the wealth, just like any other x% of the population, so the cumulative wealth distribution graph would be exactly the diagonal line and thus the area between the graph and the diagonal line would be zero. in the opposite scenario, an ultimate dictatorship where one person controls everything, the bottom x% would have exactly nothing all the way up until the last person, who would have everything; hence, the area between that curve and the diagonal line would be equal to the entire area under the diagonal line, and the coefficient would be exactly one. most real-world scenarios are in between the two. note that gini coefficients of wealth and gini coefficients of income are different things; one measures how much people have and one measures the rate at which people receive. because savings are superlinear in income, coefficients of wealth tend to be higher; the gini coefficient of wealth in the us, for example, is 0.801, and the coefficient of the world is 0.804. given that gini coefficients in the real world measure inequality of access to resources, and gini coefficients in cryptocurrency distribution arise from both inequality of resources and inequality of interest (some people care about ethereum slightly, some care about it a whole lot), 0.836 is a pretty decent result as a point of comparison, the gini coefficient of bitcoin has been measured at 0.877. the top 100 current eth holders are responsible for 45.7% of all eth, a lower percentage than the top 100 holders of the mainstream altcoins, where that statistic tends to be between 55% and 70%. of course, these last two comparisons are misleading the ethereum ecosystem has not even started to actually run, and services like exchanges which centralize control over currency units into a few wallets without centralizing legal ownership do end up artificially inflating both the gini index and the top-100 score of cryptocurrency networks that are actually live. once ethereum launches, the gini index may well prove to be impossible to accurately estimate, since large quantities of ether will be stored inside decentralized applications running arbitrary, turing-complete, and thus in many cases mathematically inscrutable, rulesets for how the ether can be withdrawn. the sale still has 28 days left to go; although we are not expecting much out of this remaining period, anything is possible. with organizational issues being wrapped up, the organization is getting ready to substantially scale up development, putting us on the fast track to finally completing the ethereum code and launching the genesis block; eta winter 2014-2015. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle exit games for evm validiums: the return of plasma 2023 nov 14 see all posts special thanks to karl floersch, georgios konstantopoulos and martin koppelmann for feedback, review and discussion. plasma is a class of blockchain scaling solutions that allow all data and computation, except for deposits, withdrawals and merkle roots, to be kept off-chain. this opens the door to very large scalability gains that are not bottlenecked by on-chain data availability. plasma was first invented in 2017, and saw many iterations in 2018, most notably minimal viable plasma, plasma cash, plasma cashflow and plasma prime. unfortunately, plasma has since largely been superseded by rollups, for reasons primarily having to do with (i) large client-side data storage costs, and (ii) fundamental limitations of plasma that make it hard to generalize beyond payments. the advent of validity proofs (aka zk-snarks) gives us a reason to rethink this decision. the largest challenge of making plasma work for payments, client-side data storage, can be efficiently addressed with validity proofs. additionally, validity proofs provide a wide array of tools that allow us to make a plasma-like chain that runs an evm. the plasma security guarantees would not cover all users, as the fundamental reasons behind the impossibility of extending plasma-style exit games to many kinds of complex applications still remain. however, a very large percentage of assets could nevertheless be kept secure in practice. this post describes how plasma ideas can be extended to do such a thing. overview: how plasma works the simplest version of plasma to understand is plasma cash. plasma cash works by treating each individual coin as a separate nft, and tracking a separate history for each coin. a plasma chain has an operator, who is responsible for making and regularly publishing blocks. the transactions in each block are stored as a sparse merkle tree: if a transaction transfers ownership of coin k, it appears in position k of the tree. when the plasma chain operator creates a new block, they publish the root of the merkle tree to chain, and they directly send to each user the merkle branches corresponding to the coins that that user owns. suppose that these are the last three transaction trees in a plasma cash chain. then, assuming all previous trees are valid, we know that eve currently owns coin 1, david owns coin 4 and george owns coin 6. the main risk in any plasma system is the operator misbehaving. this can happen in two ways: publishing an invalid block (eg. the operator includes a transaction sending coin 1 from fred to hermione even if fred doesn't own the coin at that time) publishing an unavailable block (eg. the operator does not send bob his merkle branch for one of the blocks, preventing him from ever proving to someone else that his coin is still valid and unspent) if the operator misbehaves in a way that is relevant to a user's assets, the user has the responsibility to exit immediately (specifically, within 7 days). when a user ("the exiter") exits, they provide a merkle branch proving the inclusion of the transaction that transferred that coin from the previous owner to them. this starts a 7-day challenge period, during which others can challenge that exit by providing a merkle proof of one of three things: not latest owner: a later transaction signed by the exiter transferring the exiter's coin to someone else double spend: a transaction that transferred the coin from the previous owner to someone else, that was included before the transaction transferring the coin to the exiter invalid history: a transaction that transferred the coins before (within the past 7 days) that does not have a corresponding spend. the exiter can respond by providing the corresponding spend; if they do not, the exit fails. with these rules, anyone who owns coin k needs to see all of the merkle branches of position k in all historical trees for the past week to be sure that they actually own coin k and can exit it. they need to store all the branches containing transfers of the asset, so that they can respond to challenges and safely exit with their coin. generalizing to fungible tokens the above design works for nfts. however, much more common than nfts are fungible tokens, like eth and usdc. one way to apply plasma cash to fungible tokens is to simply make each small denomination of a coin (eg. 0.01 eth) a separate nft. unfortunately, the gas costs of exiting would be too high if we do this. one solution is to optimize by treating many adjacent coins as a single unit, which can be transferred or exited all at once. there are two ways to do this: use plasma cash almost as-is, but use fancy algorithms to compute the merkle tree of a really large number of objects very quickly if many adjacent objects are the same. this is surprisingly not that hard to do; you can see a python implementation here. use plasma cashflow, which simply represents many adjacent coins as a single object. however, both of these approaches run into the problem of fragmentation: if you receive 0.001 eth each from hundreds of people who are buying coffees from you, you are going to have 0.001 eth in many places in the tree, and so actually exiting that eth would still require submitting many separate exits, making the gas fees prohibitive. defragmentation protocols have been developed, but are tricky to implement. alternatively, we can redesign the system to take into account a more traditional "unspent transaction output" (utxo) model. when you exit a coin, you would need to provide the last week of history of those coins, and anyone could challenge your exit by proving that those historical coins were already exited. a withdrawal of the 0.2 eth utxo at the bottom right could be cancelled by showing a withdrawal of any of the utxos in its history, shown in green. particularly note that the middle-left and bottom-left utxos are ancestors, but the top-left utxo is not. this approach is similar to order-based coloring ideas from colored coins protocols circa 2013. there is a wide variety of techniques for doing this. in all cases, the goal is to track some conception of what is "the same coin" at different points in history, in order to prevent "the same coin" from being withdrawn twice. challenges with generalizing to evm unfortunately, generalizing beyond payments to the evm is much harder. one key challenge is that many state objects in the evm do not have a clear "owner". plasma's security depends on each object having an owner, who has the responsibility to watch and make sure the chain's data is available, and exit that object if anything goes wrong. many ethereum applications, however, do not work this way. uniswap liquidity pools, for example, do not have a single owner. another challenge is that the evm does not attempt to limit dependencies. eth held in account a at block n could have come from anywhere in block n-1. in order to exit a consistent state, an evm plasma chain would need to have an exit game where, in the extreme case, someone wishing to exit using information from block n might need to pay the fees to publish the entire block n state on chain: a gas cost in the many millions of dollars. utxo-based plasma schemes do not have this problem: each user can exit their assets from whichever block is the most recent block that they have the data for. a third challenge is that the unbounded dependencies in the evm make it much harder to have aligned incentives to prove validity. the validity of any state depends on everything else, and so proving any one thing requires proving everything. sorting out failures in such a situation generally cannot be made incentive-compatible due to the data availability problem. a particularly annoying problem is that we lose the guarantee, present in utxo-based systems, that an object's state cannot change without its owner's consent. this guarantee is incredibly useful, as it means that the owner is always aware of the latest provable state of their assets, and simplifies exit games. without it, creating exit games becomes much harder. how validity proofs can alleviate many of these problems the most basic thing that validity proofs can do to improve plasma chain designs is to prove the validity of each plasma block on chain. this greatly simplifies the design space: it means that the only attack from the operator that we have to worry about is unavailable blocks, and not invalid blocks. in plasma cash, for example, it removes the need to worry about history challenges. this reduces the state that a user needs to download, from one branch per block in the last week, to one branch per asset. additionally, withdrawals from the most recent state (in the common case where the operator is honest, all withdrawals would be from the most recent state) are not subject to not-latest-owner challenges, and so in a validity-proven plasma chain such withdrawals would not be subject to any challenges at all. this means that, in the normal case, withdrawals can be instant! extending to the evm: parallel utxo graphs in the evm case, validity proofs also let us do something clever: they can be used to implement a parallel utxo graph for eth and erc20 tokens, and snark-prove equivalence between the utxo graph and the evm state. once you have that, you could implement a "regular" plasma system over the utxo graph. this lets us sidestep many of the complexities of the evm. for example, the fact that in an account-based system someone can edit your account without your consent (by sending it coins and thereby increasing its balance) does not matter, because the plasma construction is not over the evm state itself, but rather over a utxo state that lives in parallel to the evm, where any coins that you receive would be separate objects. extending to the evm: total state exiting there have been simpler schemes proposed to make a "plasma evm", eg. plasma free and before that this post from 2019. in these schemes, anyone can send a message on the l1 to force the operator to either include a transaction or make a particular branch of the state available. if the operator fails to do this, the chain starts reverting blocks. the chain stops reverting once someone posts a full copy of either the whole state, or at least all of the data that users have flagged as being potentially missing. making a withdrawal can require posting a bounty, which would pay for that user's share of the gas costs of someone posting such a large amount of data. schemes like this have the weakness that they do not allow instant withdrawals in the normal case, because there is always the possibility that the chain will need to revert the latest state. limits of evm plasma schemes schemes like this are powerful, but are not able to provide full security guarantees to all users. the case where they break down most clearly is situations where a particular state object does not have a clear economic "owner". let us consider the case of a cdp (collateralized debt position), a smart contract where a user has coins that are locked up and can only be released once the user pays their debt. suppose that user has 1 eth (~$2000 as of the time of this writing) locked up in a cdp with 1000 dai of debt. now, the plasma chain stops publishing blocks, and the user refuses to exit. the user could simply never exit. now, the user has a free option: if the price of eth drops below $1000, they walk away and forget about the cdp, and if the price of eth stays above, eventually they claim it. on average, such a malicious user earns money from doing this. another example is a privacy system, eg. tornado cash or privacy pools. consider a privacy system with five depositors: the zk-snarks in the privacy system keep the link between the owner of a coin coming into the system and the owner of the coin coming out hidden. suppose that only orange has withdrawn, and at that point the plasma chain operator stops publishing data. suppose also that we use the utxo graph approach with a first-in-first-out rule, so each coin gets matched to the coin right below it. then, orange could withdraw their pre-mixed and post-mixed coin, and the system would perceive it as two separate coins. if blue tries to withdraw their pre-mixed coin, orange's more recent state would supersede it; meanwhile, blue would not have the information to withdraw their post-mixed coin. this can be fixed if you allow the other four depositors to withdraw the privacy contract itself (which would supersede the deposits), and then take the coins out on l1. however, actually implementing such a mechanism requires additional effort on the part of people developing the privacy system. there are also other ways to solve privacy, eg. the intmax approach, which involves putting a few bytes on chain rollup-style together with a plasma-like operator that passes around information between individual users. uniswap lp positions have a similar problem: if you traded usdc for eth in a uniswap position, you could try to withdraw your pre-trade usdc and your post-trade eth. if you collude with the plasma chain operator, the liquidity providers and other users would not have access to the post-trade state, so they would not be able to withdraw their post-trade usdc. special logic would be required to prevent situations like this. conclusions in 2023, plasma is an underrated design space. rollups remain the gold standard, and have security properties that cannot be matched. this is particularly true from the developer experience perspective: nothing can match the simplicity of an application developer not even having to think about ownership graphs and incentive flows within their application. however, plasma lets us completely sidestep the data availability question, greatly reducing transaction fees. plasma can be a significant security upgrade for chains that would otherwise be validiums. the fact that zk-evms are finally coming to fruition this year makes it an excellent opportunity to re-explore this design space, and come up with even more effective constructions to simplify the developer experience and protect users' funds. bootstrapping an autonomous decentralized corporation, part 2: interacting with the world | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search bootstrapping an autonomous decentralized corporation, part 2: interacting with the world posted by vitalik buterin on december 31, 2013 research & development in the first part of this series, we talked about how the internet allows us to create decentralized corporations, automatons that exist entirely as decentralized networks over the internet, carrying out the computations that keep them “alive” over thousands of servers. as it turns out, these networks can even maintain a bitcoin balance, and send and receive transactions. these two capacities: the capacity to think, and the capacity to maintain capital, are in theory all that an economic agent needs to survive in the marketplace, provided that its thoughts and capital allow it to create sellable value fast enough to keep up with its own resource demands. in practice, however, one major challenge still remains: how to actually interact with the world around them. getting data the first of the two major challenges in this regard is that of input – how can a decentralized corporation learn any facts about the real world? it is certainly possible for a decentralized corporation to exist without facts, at least in theory; a computing network might have the zermelo-fraenkel set theory axioms embedded into it right from the start and then embark upon an infinite loop proving all possible mathematical theorems – although in practice even such a system would need to somehow know what kinds of theorems the world finds interesting; otherwise, we may simply learn that a+b=b+a, a+b+c=c+b+a,a+b+c+d=d+c+b+a and so on. on the other hand, a corporation that has some data about what people want, and what resources are available to obtain it, would be much more useful to the world at large. here we must make a distinction between two kinds of data: self-verifying data, and non-self-verifying data. self-verifying data is data which, once computed on in a certain way, in some sense “proves” its own validity. for example, if a given decentralized corporation is looking for prime numbers containing the sequence ’123456789′, then one can simply feed in ’12345678909631′ and the corporation can computationally verify that the number is indeed prime. the current temperature in berlin, on the other hand, is not self-verifying at all; it could be 11′c, but it could also just as easily be 17′c, or even 231′c; without outside data, all three values seem equally legitimate. bitcoin is an interesting case to look at. in the bitcoin system, transactions are partially self-verifying. the concept of a “correctly signed” transaction is entirely self-verifying; if the transaction’s signature passes the elliptic curve digital signature verification algorithm, then the transaction is valid. in theory, you might claim that the transaction’s signature correctness depends on the public key in the previous transaction; however, this actually does not at all detract from the self-verification property – the transaction submitter can always be required to submit the previous transaction as well. however, there is something that is not self-verifying: time. a transaction cannot spend money before that money was received and, even more crucially, a transaction cannot spend money that has already been spent. given two transactions spending the same money, either one could have theoretically come first; there is no way to self-verify the validity of one history over the other. bitcoin essentially solves the time problem with a computational democracy. if the majority of the network agrees that events happened in a certain order, then that order is taken as truth, and the incentive is for every participant in this democratic process to participate honestly; if any participant does not, then unless the rogue participant has more computing power than the rest of the network put together his own version of the history will always be a minority opinion, and thus rejected, depriving the miscreant of their block revenue. in a more general case, the fundamental idea that we can gleam from the blockchain concept is this: we can use some kind of resource-democracy mechanism to vote on the correct value of some fact, and ensure that people are incentivized to provide accurate estimates by depriving everyone whose report does not match the “mainstream view” of the monetary reward. the question is, can this same concept be applied elsewhere as well? one improvement to bitcoin that many would like to see, for example, is a form of price stabilization; if bitcoin could track its own price in terms of other currencies or commodities, for example, the algorithm could release more bitcoins if the price is high and fewer if the price is low – naturally stabilizing the price and reducing the massive spikes that the current system experiences. however, so far, no one has yet figured out a practical way of accomplishing such a thing. but why not? the answer is one of precision. it is certainly possible to design such a protocol in theory: miners can put their own view of what the bitcoin price is in each block, and an algorithm using that data could fetch it by taking the median of the last thousand blocks. miners that are not within some margin of the median would be penalized. however, the problem is that the miners have every incentive, and substantial wiggle room, to commit fraud. the argument is this: suppose that the actual bitcoin price is 114 usd, and you, being a miner with some substantial percentage of network power (eg. 5%), know that there is a 99.99% chance that 113 to 115 usd will be inside the safe margin, so if you report a number within that range your blocks will not get rejected. what should you say that the bitcoin price is? the answer is, something like 115 usd. the reason is that if you put your estimate higher, the median that the network provides might end up being 114.05 btc instead of 114 btc, and the bitcoin network will use this information to print more money – increasing your own future revenue in the process at the expense of existing savers. once everyone does this, even honest miners will feel the need to adjust their estimates upwards to protect their own blocks from being rejected for having price reports that are too low. at that point, the cycle repeats: the price is 114 usd, you are 99.99% sure that 114 to 116 usd will be within the safe margin, so you put down the answer of 116 usd. one cycle after that, 117 usd, then 118 usd, and before you know it the entire network collapses in a fit of hyperinflation. the above problem arose specifically from two facts: first, there is a range of acceptable possibilities with regard to what the price is and, second, the voters have an incentive to nudge the answer in one direction. if, instead of proof of work, proof of stake was used (ie. one bitcoin = one vote instead of one clock cycle = one vote), then the opposite problem would emerge: everyone would bid the price down since stakeholders do not want any new bitcoins to be printed at all. can proof of work and proof of stake perhaps be combined to somehow solve the problem? maybe, maybe not. there is also another potential way to resolve this problem, at least for applications that are higher-level than the underying currency: look not at reported market prices, but at actual market prices. assume, for example, that there already exists a system like ripple (or perhaps something based on colored coins) that includes a decentralized exchange between various cryptographic assets. some might be contracts representing assets like gold or us dollars, others company shares, others smart property and there would obviously also be trust-free cryptocurrency similar to bitcoin as well. thus, in order to defraud the system, malicious participants would not simply need to report prices that are slightly incorrect in their favored direction, but would need to push the actual prices of these goods as well – essentially, a libor-style price fixing conspiracy. and, as the experiences of the last few years have shown, libor-style price fixing conspiracies are something that even human-controlled systems cannot necessarily overcome. furthermore, this fundamental weakness that makes it so difficult to capture accurate prices without a crypto-market is far from universal. in the case of prices, there is definitely much room for corruption – and the above does not even begin to describe the full extent of corruption possible. if we expect bitcoin to last much longer than individual fiat currencies, for example, we might want the currency generation algorithm to be concerned with bitcoin’s price in terms of commodities, and not individual currencies like the usd, leaving the question of exactly which commodities to use wide open to “interpretation”. however, in most other cases no such problems exist. if we want a decentralized database of weather in berlin, for example, there is no serious incentive to fudge it in one direction or the other. technically, if decentralized corporations started getting into crop insurance this would change somewhat, but even there the risk would be smaller, since there wowuld be two groups pulling in opposite directions (namely, farmers who want to pretend that there are droughts, and insurers who want to pretend that there are not). thus, a decentralized weather network is, even with the technology of today, an entirely possible thing to create. acting on the world with some kind of democratic voting protocol, we reasoned above, it’s possible for a decentralized corporation to learn facts about the world. however, is it also possible to do the opposite? is it possible for a corporation to actually influence its environment in ways more substantial than just sitting there and waiting for people to assign value to its database entries as bitcoin does? the answer is yes, and there are several ways to accomplish the goal. the first, and most obvious, is to use apis. an api, or application programming interface, is an interface specifically designed to allow computer programs to interact with a particular website or other software program. for example, sending an http get request tohttp://blockchain.info/address/1aezym6pxy1gxiqvsrlfenjlhdjbcj4fjz?format=json sends an instruction to blockchain.info’s servers, which then give you back a file containing the latest transactions to and from the bitcoin address 1aezym6pxy1gxiqvsrlfenjlhdjbcj4fjz in a computer-friendly format. over the past ten years, as business has increasingly migrated onto the internet, the number of services that are accessible by api has been rapidly increasing. we have internet search, weather, online forums, stock trading, and more apis are being created every year. with bitcoin, we have one of the most critical pieces of all: an api for money. however, there still remains one critical, and surprisingly mundane, problem: it is currently impossible to send an http request in a decentralized way. the request must eventually be sent to the server all in one piece, and that means that it must be assembled in its entirety, somewhere. for requests whose only purpose is to retrieve public data, like the blockchain query described above, this is not a serious concern; the problem can be solved with a voting protocol. however, if the api requires a private api key to access, as all apis that automate activities like purchasing resources necessarily do, having the private key appear in its entirety, in plaintext, anywhere but at the final recipient, immediately compromises the private key’s privacy. requiring requests to be signed alleviates this problem; signatures, as we saw above, can be done in a decentralized way, and signed requests cannot be tampered with. however, this requires additional effort on the part of api developers to accomplish, and so far we are nowhere near adopting signed api requests as a standard. even with that issue solved, another issue still remains. interacting with an api is no challenge for a computer program to do; however, how does the program learn about that api in the first place? how does it handle the api changing? what about the corporation running a particular api going down outright, and others coming in to take its place? what if the api is removed, and nothing exists to replace it? finally, what if the decentralized corporation needs to change its own source code? these are problems that are much more difficult for computers to solve. to this, there is only one answer: rely on humans for support. bitcoin heavily relies on humans to keep it alive; we saw in march 2013 how a blockchain fork required active intervention from the bitcoin community to fix, and bitcoin is one of the most stable decentralized computing protocols that can possibly be designed. even if a 51% attack happens, a blockchain fork splits the network into three, and a ddos takes down the five major mining pools all at the same time, once the smoke clears some blockchain is bound to come out ahead, the miners will organize around it, and the network will simply keep on going from there. more complex corporations are going to be much more fragile; if a money-holding network somehow leaks its private keys, the result is that it goes bankrupt. but how can humans be used without trusting them too much? if the humans in question are only given highly specific tasks that can easily be measured, like building the fastest possible miner, then there is no issue. however, the tasks that humans will need to do are precisely those tasks that cannot so easily be measured; how do you figure out how much to reward someone for discovering a new api? bitcoin solves the problem by simply removing the complexity by going up one layer of abstraction: bitcoin’s shareholders benefit if the price goes up, so shareholders are encouraged to do things that increase the price. in fact, in the case of bitcoin an entire quasi-religion has formed around supporting the protocol and helping it grow and gain wider adoption; it’s hard to imagine every corporation having anything close to such a fervent following. hostile takeovers alongside the “future proofing” problem, there is also another issue that needs to be dealt with: that of “hostile takeovers”. this is the equivalent of a 51% attack in the case of bitcoin, but the stakes are higher. a hostile takeover of a corporation handling money means that the attacker gains the ability to drain the corporation’s entire wallet. a hostile takeover of decentralized dropbox, inc means that the attacker can read everyone’s files (although hopefully the files are encrypted, although in the case the attacker can still deny everyone their files). a hostile takeover of a decentralized web hosting company can lead to massive losses not just for those who have websites hosted, but also their customers, as the attacker gains the ability to modify web pages to also send off customers’ private data to the attacker’s own server as soon as each customer logs in. how might a hostile takeover be accomplished? in the case of the 501-out-of-1000 private key situation, the answer is simple: pretend to be a few thousand different servers at the same time, and join the corporation with all of them. by forwarding communications through millions of computers infected by a botnet, this is easy to accomplish without being detected. then, once you have more than half of the servers in the network, you can immediately proceed to cash out. fortunately, the presence of bitcoin has created a number of solutions, of which the proof of work used by bitcoin itself is only one. because bitcoin is a perfect api for money, any kind of protocol involving monetary scarcity and incentives is now available for computer networks to use. proof of stake, requiring each participating node to show proof that it controls, say, 100 btc is one possible solution; if that is done, then implementing a hostile takeover would require more resources than all of the legitimate nodes committed together. the 100 btc could even be moved to a multisignature address partially controlled by the network as a surety bond, both discouraging nodes from cheating and giving their owners a great incentive to act and even get together to keep the corporation alive. another alternative might simply be to allow the decentralized corporation to have shareholders, so that shareholders get some kind of special voting privileges, along with the right to a share of the profits, in exchange for investing; this too would encourage the shareholders to protect their investment. making a more fine-grained evaluation of an individual human employee is likely impossible; the best solution is likely to simply use monetary incentives to direct people’s actions on a coarse level, and then let the community self-organize to make the fine-grained adjustments. the extent to which a corporation targets a community for investment and participation, rather than discrete individuals, is the choice of its original developers. on the one hand, targeting a community can allow your human support to work together to solve problems in large groups. on the other hand, keeping everyone separate prevents collusion, and in that way reduces the likelihood of a hostile takeover. thus, what we have seen here is that very significant challenges still remain before any kind of decentralized corporation can be viable. the problem will likely be solved in layers. first, with the advent of bitcoin, a self-supporting layer of cryptographic money exists. next, with ripple and colored coins, we will see crypto-markets emerge, that can then be used to provide crypto-corporations with accurate price data. at the same time, we will see more and more crypto-friendly apis emerge to serve decentralized systems’ needs. such apis will be necessary regardless of whether decentralized corporations will ever exist; we see today just how difficult cryptographic keys are to keep secure, so infrastructure suitable for multiparty signing will likely become a necessity. large certificate signing authorities, for example, hold private keys that would result in hundreds of millions of dollars worth of security breaches if they were ever to fall into the wrong hands, and so these organizations often employ some form of multiparty signing already. finally, it will still take time for people to develop exactly how these decentralized corporations would work. computer software is increasingly becoming the single most important building block of our modern world, but up until now search into the area has been focused on two areas: artificial intelligence, software working purely on its own, and software tools working under human beings. the question is: is there something in the middle? if there is, the idea of software directing humans, the decentralized corporation, is exactly that. contrary to fears, this would not be an evil heartless robot imposing an iron fist on humanity; in fact, the tasks that the corporation will need to outsource are precisely those that require the most human freedom and creativity. let’s see if it’s possible. see also: http://bitcoinmagazine.com/7050/bootstrapping-a-decentralized-autonomous-corporation-part-i/ http://bitcoinmagazine.com/7235/bootstrapping-a-decentralized-autonomous-corporation-part-3-identity-corp/ supplementary reading: jeff garzik’s article on one practical example of what an autonomous corporation might be useful for previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum project update | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum project update posted by joseph lubin on june 5, 2014 research & development development of the ethereum project has gone very well over the months since inception.  the core activity — development of the software platform — is on track and many developers around the world are starting to build small, exploratory distributed applications on the platform, even though we are still not yet in alpha release. the 5th installment of the proof of concept series will be released soon, though many community members download the developing source code, compile it and use it on a regular basis for their various activities. other core activities include attention to legalities in different jurisdictions regarding the pre-sale, development and testing of the hot/cold wallet system, and development and testing of the sale web site.  these are well on their way to completion.  documentation regarding the sale is also nearing completion though there are still some incomplete items in the critical path that makes it difficult for us to set a sale date as yet. all of these activities are embedded in an ongoing base of support and communications activities.  these have grown extensive due to the global popularity of the project.  our twitter followers number in excess of 6000.  the number of global meet-ups that focus on ethereum exclusively or significantly currently numbers 58 groups in 49 different cities, spread over 19 countries.  13,300 people have subscribed to our newsletter.  and the  various discussions on reddit/r/etherum and forum.ethereum.org are active. this project is agile and fast moving and has benefitted from several leadership reorganizations since inception. it is anticipated that this fluidity will continue as each of us are dedicated remaining lean and agile and to doing what is best for the project at any given time. stephan tual and taylor gerring have been on the project and essential to it since the early days.  though they had significant input already, this has been formalized and they are now part of the leadership group. the most recent phase of business activity involved strong attention to the legalities of the sale and charles hoskinson was ideal to lead that effort.  but that effort is now substantially complete and charles will be moving on to other activities in the space. please stay tuned to this channel as we will continue to release information regarding software development activities and details of the pre-sale for absorption and feedback. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements visions, part 1: the value of blockchain technology | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search visions, part 1: the value of blockchain technology posted by vitalik buterin on april 13, 2015 research & development one of the questions that has perhaps been central to my own research in blockchain technology is: ultimately, what is it even useful for? why do we need blockchains for anything, what kinds of services should be run on blockchain-like architectures, and why specifically should services be run on blockchains instead of just living on plain old servers? exactly how much value do blockchains provide: are they absolutely essential, or are they just nice to have? and, perhaps most importantly of all, what is the "killer app" going to be? over the last few months, i have spent a lot of time thinking about this issue, discussing it with cryptocurrency developers, venture capital firms, and particularly people from outside the blockchain space, whether civil liberties activists, people in the finance and payments industry or anywhere else. in the process of this, i have come to a number of important, and meaningful, conclusions. first, there will be no "killer app" for blockchain technology. the reason for this is simple: the doctrine of low-hanging fruit. if there existed some particular application for which blockchain technology is massively superior to anything else for a significant portion of the infrastructure of modern society, then people would be loudly talking about it already. this may seem like the old economics joke about an economist finding a twenty dollar bill on the ground and concluding it must be fake because otherwise it would already have been taken, but in this case the situation is subtly different: unlike the dollar bill, where search costs are low and so picking up the bill makes sense even if there is only a 0.01% chance it is real, here search costs are very high, and plenty of people with billions of dollars of incentive have already been searching. and so far, there has been no single application that anyone has come up with that has seriously stood out to dominate everything else on the horizon. in fact, one can quite reasonably argue that the closest things that we will ever have to "killer apps" are precisely those apps that have already been done and recited and sensationalized ad nauseam: censorship resistance for wikileaks and silk road. silk road, the online anonymous drug marketplace that was shut down by law enforcement in late 2013, processed over $1 billion in sales during its 2.5 years of operations, and while the payment-system-orchestrated blockade against wikileaks was in progress, bitcoin and litecoin donations were responsible for the bulk of its revenue. in both cases the need was clear and the potential economic surplus was very high before bitcoin, you would have no choice but to buy the drugs in person and donate to wikileaks by cash-in-the-mail, and so bitcoin provided a massive convenience gain and thus the opportunity was snatched up almost instantly. now, however, that is much less the case, and marginal opportunities in blockchain technology are not nearly such easy grabs. total and average utility does this mean, however, that blockchains have hit their peak utility? most certainly not. they have hit peak necessity, in the sense of peak utility per user, but that is not the same thing as peak utility. although silk road was indispensable for many of the people that used it, even among the drug-using community it's not indispensable in general; as much as it befuddles this particular author how ordinary individuals are supposed to get such connections, most people have somehow found "a guy" that they know that they can purchase their weed from. interest in smoking weed at all seems to strongly correllate with having easy access to it. hence, in the grand scheme of things, silk road has only had a chance to become relevant to a very niche group of people. wikileaks is similar; the set of people who care about corporate and governmental transparency strongly enough to donate money to a controversial organization in support of it is not very large compared to the entire population of the world. so what's left? in short, the long tail. so what is the long tail? this is where it gets hard to explain. i could provide a list of applications that are included in this "long tail" of applications; however, blockchains are not indispensable, and do not even offer extremely strong fundamental advantages for each one. for each individual case, an advocate of either the "blockchain applications are overrated, it's the bitcoin currency that matters" or the "blockchain tech as a whole is useless" position can quite reasonably come up with a way to implement the scheme just as easily on a centralized server, replace blockchain governance with a legal contract, and apply whatever other replacements to turn the product into something much more similar to a traditional system. and on that point, they would be completely correct: for that particular use case, blockchains are not indispensable. and that's the whole point: those applications are not at the top of the distribution, up there with wikileaks and silk road; if they were, they would have been implemented already. in the long tail, blockchains are not necessary; they are convenient. they are simply marginally better than the next available tool for the job. and yet, because these applications are much more mainstream, and can benefit hundreds of millions of users, the total gain to society (which can be seen from the area on the above chart) is much larger. perhaps the best analogy to this line of reasoning is to ask the following rhetorical question: what is the killer app of "open source"? open source has clearly been a very good thing for society, and it is being used for millions of software packages around the world, but nevertheless it is still hard to answer the question. and the reason is the same: there is no killer app, and the list of applications has a very very long tail basically, just about every kind of software imaginable, with particular emphasis on lower-level libraries that end up reused by millions of projects many times over and critical cryptographic security libraries. blockchains, redefined... again now, what are the specific benefits of blockchains that make the long tail worthwhile? to start off, let me provide the current description that i use of what a blockchain is: a blockchain is a magic computer that anyone can upload programs to and leave the programs to self-execute, where the current and all previous states of every program are always publicly visible, and which carries a very strong cryptoeconomically secured guarantee that programs running on the chain will continue to execute in exactly the way that the blockchain protocol specifies. notice that this definition does not: use financially-charged terms like "ledger", "money" or "transactions", or indeed any terms geared toward a particular use case mention any particular consensus algorithm, or indeed mention anything about the technical properties of how a blockchain works (except for the fact that it's "cryptoeconomic", a technical term roughly meaning "it's decentralized, it uses public key cryptography for authentication, and it uses economic incentives to ensure that it keeps going and doesn't go back in time or incur any other glitch") make a restriction to any particular type of state transition function the one thing that the definition does well is explain what a blockchain does, and it explains it in such a way that any software developer will be able to fairly clearly have at least an intuitive grasp of its value proposition. now, in practice, sometimes the programming language that the programs run in is very restrictive; bitcoin's language can be seen as requiring a sequence of destroy coin: <txid> <index> <scriptsig> statements followed by a sequence of create coin: <scriptpubkey> <value> statements, where scriptpubkey is a restricted mathematical formula, scriptsig must be a satisfying variable assignment to the formula (eg. {x = 5, y = 7} satisfies 2 * x y = 3), and an attempt to destroy a nonexistent coin or destroy a coin without supplying a valid scriptsig for that coin's scriptpubkey, or an attempt to create more coin value than you destroyed, returns an error. other programming languages, on the other hand, can be much more expressive. it's up to the software developer to analyze what programming language is right for their task, much like it is a software developer's task today to decide between python, c++, nodejs and malbolge. the one thing that the definition emphasizes extremely well is that blockchains are not about bringing to the world any one particular ruleset, whether it's a currency with a fixed-supply monetary policy, a name registry with a 200-day re-registration time, a particular decentralized exchange design or whatever else; rather, they're about creating the freedom to create a new mechanism with a new ruleset extremely quickly and pushing it out. they're lego mindstorms for building economic and social institutions. this is the core of the more moderate version of the "it's the blockchain that's exciting, not the currency" position that is so prevalent in mainstream industry: it is indeed true that currency is necessary to make cryptoeconomic blockchains work (although not blockchain-like data structures following the stellar subjective consensus model), but the currency is there simply as economic plumbing to incentivize consensus participation, hold deposits and pay transaction fees, not as the center-stage point of speculative mania, consumer interest and excitement. now, why are blockchains useful? to summarize: you can store data on them and that data is guaranteed to have a very high degree of availability you can run applications on them and be guaranteed an extremely high uptime you can run applications on them, and be guaranteed an extremely high uptime going very far into the future you can run applications on them, and convince your users that the application's logic is honest and is doing what you are advertising that it does you can run applications on them, and convince your users that your application will remain working even if you lose interest in maintaining it, you are bribed or threatened to manipulate the application state in some way, or you acquire a profit motive to manipulate the application state in some way you can run applications on them, and give yourself the backdoor key if it is absolutely necessary, but put "constitutional" limiations on your use of the key for example, requiring a software update to pass through a public one-month waiting period before it can be introduced, or at the very least immediately notifying users of application updates you can run applications on them, and give a backdoor key to a particular governance algorithm (eg. voting, futarchy, some complicated multicameral parliament architecture), and convince your users that the particular governance algorithm in question is actually in control of the application you can run applications on them, and those applications can talk to each other with 100% reliability even if the underlying platform has only 99.999% reliability multiple users or companies can run applications on them, and those applications can interact with each other at extremely high speed without requiring any network messages, while at the same time ensuring that each company has total control over its own application you can build applications that very easily and efficiently take advantage of the data produced by other applications (eg. combining payments and reputation systems is perhaps the largest gain here) all of those things are valuable indirectly to billions of people around the world, potentially particularly in regions of the world where highly developed economic, financial and social infrastructure currently simply does not work at all (though technology will often need to be combined with political reforms to solve many of the problems), and blockchains are good at providing these properties. they are particularly obviously valuable in finance, as finance is perhaps the most simultaneously computationally and trust-intensive industry in the world, but they are also valuable in many other spots in internet infrastructure. there do exist other architectures that can also provide these properties, but they are slightly to moderately less good than blockchains are. gavin wood has started describing this ideal computing platform as "the world computer" a computer the state of which is shared among everyone and which a very large group of people, which anyone is free to join, are involved in maintaining. base layer infrastructure like open source, by far the largest opportunity for gains out of blockchain technology are out of what can be called "base-layer infrastructure" services. base-layer infrastructure services, as a general category, are characterized by the following properties: dependency there exist many other services that intimately depend on the base-layer service for functionality high network effects there are substantial benefits from very large groups of people (or even everyone) using the same service high switching costs it is difficult for an individual to switch from one service to the other note that one concern that is not in there is any notion of raw "necessity" or "importance"; there can be fairly unimportant base layers (eg. rss feeds) and important non-base-layers (eg. food). base-layer services have existed ever since even before the dawn of civilization; in the so-called "caveman days" the single most important base-layer service of all was language. in somewhat more recent times, the primary examples became roads, the legal system and postal and transportation systems, in the 20th century we added telephone networks and financial systems, and at the end of the millennium emerged the internet. now, however, the new base-layer services of the internet are almost entirely informational: internet payment systems, identity, domain name systems, certificate authorities, reputation systems, cloud computing, various kinds of data feeds, and perhaps in the near future prediction markets. in ten years time, the highly networked and interdependent nature of these services may make it such that it is harder for individuals to switch from one system to another than it is for them to even switch which government they are living under and that means that making sure that these services are built correctly and that their governance process does not put a few private entities in positions of extreme power is of utmost importance. right now, many of these systems are built in a highly centralized fashion, and this is in part simply due to the fact that the original design of the world wide web failed to realize the importance of these services and include defaults and so, even today, most websites ask you to "sign in with google" or "sign in with facebook", and certificate authorities run into problems like this: "a solo iranian hacker on saturday claimed responsibility for stealing multiple ssl certificates belonging to some of the web's biggest sites, including google, microsoft, skype and yahoo. early reaction from security experts was mixed, with some believing the hacker's claim, while others were dubious. last week, conjecture had focused on a state-sponsored attack, perhaps funded or conducted by the iranian government, that hacked a certificate reseller affiliated with u.s.-based comodo. on march 23, comodo acknowledged the attack, saying that eight days earlier, hackers had obtained nine bogus certificates for the log-on sites of microsoft's hotmail, google's gmail, the internet phone and chat service skype and yahoo mail. a certificate for mozilla's firefox add-on site was also acquired." why shouldn't certificate authorities be decentralized at least to the point of an m-of-n system again? (note that the case for much more widespread use of m-of-n is logically separable from the case for blockchains, but blockchains happen to be a good platform to run m-of-n on). identity let us take a particular use case, "identity on the blockchain", and run with it. in general, what do you need in order to have an identity? the simplest answer is one we already know: you need to have a public and private key. you publish the public key, which becomes your id, and you digitally sign every message you send with your private key, allowing anyone to verify that those messages were produced by you (where, from their point of view, "you" means "the entity that holds that particular public key"). however, there are a few challenges: what happens if your key gets stolen, and you need to switch to a new one? what happens if you lose your key? what if you want to refer to other users by their names, and not just a random 20-byte string of cryptographic data? what if you want to use a more advanced approach for security such as multisig, and not just a single key? let us try solving these challenges one-by-one. we can start off with the fourth. a simple solution is this: instead of requiring one particular cryptographic signature type, your public key becomes a program, and a valid signature becomes a string that, when fed into the program together with the message, returns 1. theoretically, any single-key, multi-key or whatever other kind of ruleset can be encoded into such a paradigm. however, this has a problem: the public keys will get too long. we can solve this by putting the actual "public key" into some data store (eg. a distributed hash table if we want decentralization) and using the hash of the "public key" as the user's id. this does not yet require blockchains although, in the latest designs, in the limit scalable blockchains are really not that different in design from dhts and so it is entirely possible that, in ten years time, every kind of decentralized system used for anything will accidentally or intentionally converge into some kind of scalable blockchain. now, consider the first problem. we can think of this as the certificate revocation problem: if you want to "revoke" a particular key, how do you ensure that it gets around to everyone who needs to see it? this by itself can once again be solved by a distributed hash table. however, this leads to the next problem: if you want to revoke a key, what do you replace it with? if your key is stolen, you and the attacker both have it, and so neither of you can be convincingly more authoritative. one solution is to have three keys, and then if one gets revoked then require a signature from two or all of them to approve the next key. but this leads to a "nothing at stake" problem: if the attacker eventually manages to steal all three of your keys from some point in history, then they can simulate a history of assigning a new key, assigning further new keys from there, and your own history is no longer more authoritative. this is a timestamping problem, and so here blockchains can actually help. for the second problem, holding multiple keys and reassigning also works reasonably well and here, blockchains are not needed. in fact, you do not need to re-assign; with clever use of secret sharing you can actually recover from key losses simply by keeping your key in "shards", such that if you lose any single shard you can always use secret sharing math to simply recover it from the others. for the third problem, blockchain-based name registries are the simplest solution. however, in practice most people are not well-equipped to securely store multiple keys, and there are always going to be mishaps, and often centralized services play an important role: helping people get their accounts back in the event of a mistake. in this case, the blockchain-based solution is simple: social m-of-n backup. you pick eight entities; they may be your friends, your employer, some corporation, nonprofit or even in the future a government, and if anything goes wrong a combination of five of them can recover your key. this concept of social multi-signature backup is perhaps one of the most powerful mechanisms to use in any kind of decentralized system design, and provides a very high amount of security very cheaply and without relying on centralized trust. note that blockchain-based identity, particularly with ethereum's contract model, makes all of this very easy to program: in the name registry, register your name and point it at a contract, and have that contract maintain the current main key and backup keys associated with the identity as well as the logic for updating them over time. an identity system, safe and easy-to-use enough for grandma, done without any individual entity (except for you!) in control. identity is not the only problem that blockchains can alleviate. another component, intimately tied up with identity, is reputation. currently, what passes for "reputation systems" in the modern world are invariably either insecure, due to their inability to ensure that an entity rating another entity actually interacted with them, or centralized, tying reputation data to a particular platform and having the reputation data exist under that platform's control. when you switch from uber to lyft, your uber rating does not carry over. a decentralized reputation system would ideally consist of two separate layers: data and evaluation. data would consist of individuals making independent ratings about others, ratings tied to transactions (eg. with blockchain-based payments one can create an open system such that you can only give merchants a rating if you actually pay them), and a collection of other sources, and anyone can run their own algorithm to evaluate their data; "light-client friendly" algorithms that can evaluate a proof of reputation from a particular dataset quickly may become an important research area (many naive reputation algorithms involve matrix math, which has nearly cubic computational complexity in the underlying data and so is hard to decentralize). "zero-knowledge" reputation systems that allow a user to provide some kind of cryptographic certificate proving that they have at least x reputation points according to a particular metric without revealing anything else are also promising. the case of reputation is interesting because it combines together multiple benefits of the blockchain as a platform: its use as a data store for identity its use as a data store for reputational records inter-application interoperability (ratings tied to proof of payment, ability for any algorithm to work over the same underlying set of data, etc) a guarantee that the underlying data will be portable going into the future (companies may voluntarily provide a reputation certificate in an exportable format, but they have no way to pre-commit to continuing to have that functionality going into the future) the use of a decentralized platform more generally to guarantee that the reputation wasn't manipulated at the point of calculation now, for all of these benefits, there are substitutes: we can trust visa and mastercard to provide cryptographically signed receipts that a particular transaction took place, we can store reputational records on archive.org, we can have servers talk to each other, we can have private companies specify in their terms of service that they agree to be nice, and so forth. all of these options are reasonably effective, but they are not nearly as nice as simply putting everything out into the open, running it on "the world computer" and letting cryptographic verification and proofs do the work. and a similar argument can be made for every other use case. cutting costs if the largest value from blockchain technology comes at the long tail, as this thesis suggests, then that leads to an important conclusion: the per-transaction gain from using a blockchain is very small. hence, the problem of cutting costs of consensus and increasing blockchain scalability becomes paramount. with centralized solutions, users and businesses are used to paying essentially $0 per "transaction"; although individuals looking to donate to wikileaks may be willing to pay even a fee of $5 to get their transaction through, someone trying to upload a reputation record may well only be willing to pay a fee of $0.0005. hence, the problem of making consensus cheaper, both in the absolute sense (ie. proof of stake) and in the per-transaction sense (ie. through scalable blockchain algorithms where at most a few hundred nodes process each transaction), is absolutely paramount. additionally, blockchain developers should keep in mind that the last forty years of software development has been a history of moving to progressively less and less efficient programming languages and paradigms solely because they allow developers to be less experienced and lazier, and similarly work to design blockchain algorithms that work around the principle that developers are really not going to be all that smart and judicious about what they put on the blockchain and what they keep off though a well-designed system of transaction fees will likely lead to developers naturally learning most of the important points through personal experience. hence, there is substantial hope for a future that can be, to a substantial degree, more decentralized; however, the days of easy gains are over. now is the time for a much harder, and longer, slog of looking into the real world, and seeing how the technologies that we have built can actually benefit the world. during this stage, we will likely discover that at some point we will hit an inflection point, where most instances of "blockchain for x" will be made not by blockchain enthusiasts looking for something useful to do, coming upon x, and trying to do it, but rather by x enthusiasts who look at blockchains and realize that they are a fairly useful tool for doing some part of x. whether x is internet of things, financial infrastructure for the developing world, bottom-up social, cultural and economic institutions, better data aggregation and protection for healthcare, or simply controversial charities and uncensorable marketplaces. in the latter two cases, the inflection point has likely already hit; many of the original crowd of blockchain enthusiasts became blockchain enthusiasts because of the politics. once it hits in the other cases, however, then we will truly know that it has gone mainstream, and that the largest humanitarian gains are soon to come. additionally, we will likely discover that the concept of "the blockchain community" will cease to be meaningful as any kind of quasi-political movement in its own right; if any label applies at all, "crypto 2.0" is likely to be the most defensible one. the reason is similar to why we do not have a concept of "the distributed hash table community", and "the database community", while existent, is really simply a set of computer scientists who happen to specialize in databases: blockchains are just one technology, and so ultimately the greatest progress can only be achieved by working on combination with a whole set of other set of decentralized (and decentralization-friendly) technologies: reputation systems, distributed hash tables, "peer-to-peer hypermedia platforms", distributed messaging protocols, prediction markets, zero-knowledge proofs and likely many more that have not yet been discovered. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements devgrants: here to help | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search devgrants: here to help posted by ðξvgrants team on april 7, 2015 ecosystem support program few would argue against the fact that ethereum has been gifted with one of the most diverse and thoughtful communities ever gathered around any open source project. fueled by what can only be described as impassioned determination, developers from all walks of life have risked their time and credibility to pioneer in this radical and largely unknown domain of innovation. indeed, even before genesis, ethereum already boasts hundreds of projects covering everything from self-stabilizing currency to potential replacements for government services. today we are launching a new program to support and thank those who continue to take this journey with us. introducing ðξvgrants. ðξvgrants is a program that offers funding for contributions to both the ethereum platform, and projects based on ethereum. its goals are threefold: to provide developers interested in contributing to the ethereum ecosystem the opportunity to spend significant time on their project, in order to bring it to completion. to extend the codebase with useful components that are not the main focus of ξthdev, but which would be very valuable to users of ethereum generally. to increase our outreach to other communities and the general public. ðξvgrants will focus exclusively on giving a boost to efforts already in progress. in other words, if you have not already begun work on your project, you should do so before attempting to apply. note that although ξthdev does not bias against applications from businesses or developers who may wish to profit from the fruits of ðξvgrant-funded projects, ðξvgrants should not be considered a funding platform for new business endeavors. only those applicants creating an open source public good for public benefit, will be considered. what ðξvgrants is not it is not a venture capital programme. we will not "invest" or "grant" funding for projects which do not have on-going development. it is not a bounty programme. we do not seek to describe specific goals in advance nor do we wish to attribute value to them. it is not a ðapp seed funding programme. at this point in our development process, most ðapps score near the bottom of critical value for our development process. to be eligable, you must: be already working on a project; reasonably be delivering value to critical components of the ethereum core software, its development process or key parts of the ecosystem (common services, apis, abis). grant funding may be used to help deliver or maximise value of personal contributions, e.g.: payment of professional services or equipment; payment of expenses incurred as part of collaborative workflow; payment to help with subsistence costs of project members (generally when taking time from employment). the application process. parties interested in receiving funding should get in touch with the ðξvgrants administrator (grants@ethereum.org), submitting three items: a proposal that outlines the intended work, a project plan that details timelines and estimated working hours, and a proposed methodology for reporting on the progress and outcomes of the project. we additionally ask that you include one or more means of realtime contact – xmpp/telegram/skype id, irc nick, phone number, etc – and your time zone. ξthdev will make a best effort to evaluate new proposals within 2 weeks of receipt. approved applicants will be requested to sign a contract with ξthdev, and to accelerate development as soon as possible. the contract is a legal contract between the successful applicant and ξthdev, and we recommend that all applicants review the terms carefully, prior to submitting their application. how will evaluation work? initial evaluation the initial evaluation will be conducted by the ðξvgrants administrator, consisting of a review of the application, a high-level investigation into the state of the project, and an online interview with one or more of the project developers. this evaluation will concern itself with the social and technical merits of the project, including its broader applicability and benefit to the ethereum ecosystem, the general character of its developers, and the progress made so far. final evaluation final evaluations will be conducted by the ðξvgrants board, a team consisting of founding members of ethereum, plus the ðξvgrants administrator. this evaluation will review the summarized results of the initial evaluations of the previous two weeks. applicants may or may not be invited to parts of this discussion for the purposes of further review and examination. what should a proposal describe? while we understand that each project will have unique properties to underline and stress, all applications will benefit from addressing the following unambiguously: the project's goal and scope. the targeted deliverables for the work period. timelines and estimated working hours. the amount of funding requested. how the project benefits the ethereum ecosystem. the methodology for reporting on progress and achievement. how much funding can be requested, and how does payment work? while exceptions can be made in certain cases, ðξvgrants is targeting needs ranging from usd 1,000to1,000 to 1,000to10,000 per project, or just enough to cover a few months of living expenses for one or more developers. ξthdev can offer disbursement in two different ways, depending on the amount requested, the length of the project’s proposed development period, and the work done upfront: option one: half of the total amount approved at the start of the project, and half upon completion. option two: multiple payments, based on milestone achievements. the desired method can and should be requested in the application, but ξthdev ultimately reserves the right to choose. what about transparency? all projects approved by ðξvgrants will be made public in both name and application, on the forthcoming ðξvgrants web site and on the @devgrants twitter account. in most cases, any project funded will already have a publicly-accessible repository, and in all cases must adhere to one or more specified open source licenses. in conclusion again, we thank you, the community, for all of your development efforts, critiques, and encouragement. it is our hope that this program will lend a small but helpful hand to your remarkable work, and we look forward to reading your applications in full. comments, questions and applications should be directed to the ðξvgrants administrator at grants@ethereum.org. interested parties may also follow @devgrants on twitter for updates about funded projects, evaluation policy changes, and more. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements launching the ether sale | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search launching the ether sale posted by vitalik buterin on july 22, 2014 organizational first of all, i would like to thank the community, and especially those close to the project who have in many cases abandoned their jobs to dedicate their time to it, for their extreme patience regarding the launch of the ether sale. we have been promising that the sale would arise in two weeks for six months, and many team members have endured substantial hardships because of expectations that we set regarding when we would be able to provide funding. we certainly miscalculated the sheer difficulty of navigating the relevant legal processes in the united states and switzerland, as well as the surprisingly intricate technical issues surrounding setting up a secure sale website and cold wallet system. however, the long wait is finally over, and we are excited to announce the launch of the sale. in the intervening six months, our time has been spent on the following: developing the ethereum clients, to the point where we now have four full implementations of the protocol, of which two are fully compatible and the other two nearly so. originally, the intent was for only three implementations; however, a member of the community (roman mandeleil) has independently come forward with a functional java client. poc5 has been released today. developing the ethereum programming languages and compilers, to the point where we have moderately stable versions of serpent, lll and mutan. developing the ethereum protocol. major changes between february and june include: the transactions-pay-for-computation system, where a transaction specifies the amount of "gas" that it is allowed to consume and pre-pays a fee for it. originally, contracts paid for computation, requiring every contract to have a security "wrapper" mandating a minimum fee to avoid balance-starvation attacks. the concept of a recursive virtual machine, where contracts can "call" other contracts like functions and receive "return values" in response. this allows for a high level of generalization and code reuse in the blockchain, cutting down bloat. a 10 to 20-second block time, down from 60s. this will be added in poc6. the concept of "micro-chains", where the top-level block header will only contain pow-relevant information, reducing the minimum storage and bandwidth requirements for a light node by 4x (although the fast block time will once again increase these requirements by 4x). this will be added in poc6. the placement of transactions into a merkle tree, allowing for the development of a light client protocol for verification of individual transaction execution. the concept of memory as a byte array, making it easier to port existing compilers to compiling ethereum scripts. mid-stage developments in the mining algorithm thanks primarily to the work of vlad zamfir. more details should be released soon. developing an increasingly large repertoire of testing facilities in order to help ensure cross-client compatibility. setting up our legal organizational structure in switzerland. legal due-diligence processes in the united states.   ether will be purchasable directly through our website at https://ethereum.org. there are a number of documents providing more details on the sale as well, including a development plan (đξv plan), roadmap, intended use of revenue document, terms and conditions of the ethereum genesis sale, ether product purchase agreement, and updated white paper and yellow paper. particularly note the following: ether will not be usable or transferable until the launch of the genesis block. that is to say, when you buy ether and download your wallet, you will not be able to do anything with it until the genesis block launches. the price of ether is initially set to a discounted price of 2000 eth per btc, and will stay this way for 14 days before linearly declining to a final rate of 1337 eth per btc. the sale will last 42 days, concluding at 23:59 zug time september 2. the expected launch time for the genesis block is winter 2014-2015. this date has been continually pushed back in large part due to funding delays, but development should soon start to proceed much more quickly. ether is a product, not a security or investment offering. ether is simply a token useful for paying transaction fees or building or purchasing decentralized application services on the ethereum platform; it does not give you voting rights over anything, and we make no guarantees of its future value. we are not blocking the us after all. yay. it is very likely that alternative versions of ethereum (“alt-etherea” or “forks”) will be released as separate blockchains, and these chains will compete against the official ethereum for value. successfully purchasing eth from the genesis sale only guarantees that you will receive eth on the official network, not any forks. the terms and conditions of the ethereum genesis sale, and the ether product purchase agreement, are the only authoritative documents regarding the sale. any statements that we have made up to this point are null and void.  the ethereum.org website, the blog posts since taylor gerring's recent howto and other posts we have made on various forums including our forum and reddit are to be treated as informal communications designed to keep the community updated on what is happening with ethereum. the annual issuance rate is 0.26x the initial quantity of ether sold, reduced from the 0.50x that had been proposed in january. we may choose later on to adopt alternative consensus strategies, such as hybrid proof of stake, so future patches may reduce the issuance rate lower. however, we make absolutely no promises of this, except that the issuance rate will not exceed 26.00% per annum of the quantity of ether sold in the genesis sale. there are two endowment pools, each 0.099x the initial quantity of ether sold, that will be allocated in the first case to early contributors to the project and in the second case to a long-term endowment to our non-profit foundation (see sale docs for more info). this is reduced from our proposal in april of 0.075x to early contributors and 0.225x to a foundation. we reserve the right to use up to 5000 btc from the pre-sale while the pre-sale is running in order to speed up development. we may or may not take this option depending on community response; if done at all it will be done transparently. if you are eligible for backpay or a share of the early contributor pool, you will be contacted soon. do not refresh the page while using the ether sale website. if you purchase ether, do not lose your wallet file or your password or you will never be able to access your ether. additionally, make sure you download your wallet in the first place. the exodus address is 36prz1khympqsyaqxsg8vwbuiq2eogxlo2; however, do not send btc to this address directly, since transactions that are not properly formatted will be invalid and treated as gifts to ethereum switzerland gmbh. use the sale application on the website. unlike some other projects, however, paying with btc using our sale application from a coinbase account or an exchange account poses no problems. the btc address that you send from does not matter. for advanced users and those who are uncomfortable with purchasing through a website, a python library for making purchases is also available at https://github.com/ethereum/pyethsaletool. the address is a 3-of-4 cold storage, multisig setup, between four centers which are themselves protected via mechanisms such as m-of-n shamir's secret sharing. check this transaction on blockchain.info, input 4, to see proof that we are capable of spending funds from the address. we will be releasing a more detailed description of the cold storage policy soon, although strategically important variables (eg. precise locations of the sites, passwords, combinations to safes) will unfortunately be kept secret. more information will be released throughout the sale, and we will soon have an ama on reddit. we look forward to continuing this exciting journey together! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements another ethereum ðξv update | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search another ethereum ðξv update posted by gavin wood on june 15, 2015 research & development been a while, i know, but then these are rather busy days. i'm writing this from a starbucks in shanghai sitting behind the great firewall, able to peep out only occasionally. i've been in asia with marek for the last couple of weeks, mainly for meetups, workshops and technical meetings. during this time, we've seen the release, struggle and survival of the olympic testnet, a very clear signal that a multi-client system would be far superior to the present monoculture. we've seen the beginning of the second external go audit, mainly for checking of regressions but also to offer comments and criticisms on aspects that have changed in the meantime such as go's network layer and sync strategy. we've seen mix and solidity continue to progress and mature. and through ongoing development and the progression of the apis we've seen all clients become cleaner, faster and more resilient. exchanges we've been continuing to work with exchanges, now marek and konstantin both provide substantial technical support, helping the coders from the exchanges understand the wide differences between using standard cryptocurrency apis and ethereum's smart-contract-based approach. in order to provide better support for them and maximise adoption, we've designed and introduced a middleware json rpc proxy allowing a much more familiar interface to ethereum for exchanges. c++ on the c++ side, we have introduced pv61, a new ethereum network protocol version (backwards compatible so as not to hamper the frontier release schedule for go) providing super-fast parallel hash-chain downloading and much more resilience to dirty hash-chain attacks. arkadiy is helping code that up for c++ with the hope that it can be worked into go before homestead. solidity, meanwhile, has progressed substantially: solidity can now structurally predict gas usage, optimise for storage access and code size very well, and support internal-function calls with dynamic types, together with a whole host of other features. liana continues hacking on that codebase together with christian. mix, under the guidance of aeron and jutta with avsa providing visual design insights and yann hacking, is seeing substantial ux polish and refactoring, with the general target being to have it be an ultra-usable, yet powerful, ide for contract development. ðapps may now be deployed to the network in a simple 3-stage process, and the state/blockchain management layer is becoming far improved, featuring a single-pane view from which you can easily see all transactions, calls and logs and propagate transactions between chain-refreshes easily. alex continues to work on the low-level network layer, now being helped by vlad. we had a summit in zug 3 weeks ago where i laid out the plans for our libp2p networking layer, and how the whisper and ethereum protocols fit into that. our first full-time developer on whisper, vlad will continue hacking at libp2p and whisper, making the protocol more resilient to various ethereum networking attacks whilst continuing the r&d for the p2p asynchronous non-endpoint-specific signalling protocol. pawel, working from warsaw, has been continuing on the jit evm and evm libraries, optimising stack usage and helping fix the stack depth issues we were seeing on mac os x. vladimir meanwhile continues his work on testing and marian on the netstats page which has recently proved its utility in more ways than one. those who have been experimenting with mining setups on the olympic testnet will probably recognise lefteris's work in taking over the ethash implementation, and updating, refactoring and generally supporting tim hughes' opencl implementation. special thanks to each of the members of the community who have helped one way or another with the development of this code. you know who you are (-: christoph, meanwhile, is now working on test coverage for the core classes, as well as helping fix various issues in the core. aside from working with the mix teams, arkadiy and marek, i've been coding up the new c++ wallet/secret store code (a revamped format and compatibility with go and python), optimising the core (import times for at least one gas-heavy block were recently measured as being ~10x faster than python and ~3x faster than go), bug fixing across the core, implementation of the new 'ethvm' binary for running the evm in standalone mode and the 'ethkey' binary for wallet management and, most recently, coding up a much better interactive console interface, similar to go's javascript console. i'm also hoping to get a much better blockchain download diagnostic display going soon in alethzero. auditing the internal audit of the c++ codebase has begun, with christoph working on tests for all core classes and arkadiy helping with some of the core optimisations. the secretstore and wallet classes have already been audited by christian, thereby giving credibility that the c++ client is reasonably secure for managing private keys. the c++ external audit will begin in three short weeks. since the substantial failure of the testnet caused by issues in only one of the three clients, including issues that survived two auditing procedures, it has become clear to us that auditing, external or internal, is no silver bullet. while we are committed to providing the very best software and will continue with our auditing programme, please let's be clear: there is nothing magical about the go client having gone through an auditing process. we are aiming to have all clients through at least a basic auditing process by the homestead release and i see no reason to use the go client over other clients (c++ or python) at this stage, nor, for the vast majority of users, during frontier. no clients come with any guarantees. indeed there's the argument that minority clients, being a smaller target, are less likely to be attacked. onwards we continue our preparations for the frontier release. while we're still uncertain of the precise release date, we are becoming increasingly happy at the resilience of the olympic testnet. as the olympic testnet's failure was ongoing, the adversity caused some reflection over how we might mitigate such problems in the future. the depth and duration of the consensus failure can, roughly speaking, be put down to two problems: firstly that there was a bug in the go codebase causing it to accept invalid blocks (in this case, blocks with an invalid proof-of-work nonce); secondly that there was a huge problem with upgrading the network since miners continued to mine on the 'bad' chain and were slow to upgrade their nodes so that they mined on the correct chain. essentially, the first was a forensic problem and the second an problem of organisation. to make sure this is never so deep nor broad in the future, i designed two new protocols: the bad chain canary and the bad block reporting api. the bad chain canary is a simple contract. it is controlled by a single key (though there may be several of them) and sits quietly until poked by its owner. i will be one such owner, vitalik and jeff two others and there will likely be fourth, christoph. when poked, it is given a blockhash and block number pair; this pair refers to a recent block on a bad chain. if they match the chain that the contract thinks it is sitting on, it puts itself in the "bad chain" state. this allows the ethereum core devs to inform everyone who is running a client on a bad chain (i.e. one that does not conform to the yellow paper due to a bug) that it is such. the default in clients (easily overridden) is not to mine on such chains (since to do so would be a waste of energy), but rather to inform the user that they should upgrade at their earliest opportunity. the bad block reporting api is a simple json rpc that allows a number of our nodes to run in a 'sentinel' mode. in this mode, if they come across a bad block, they automatically compile a standardised report of what went wrong including expected transaction receipts, vm traces, proof-of-work and block error information. this report is then instantly sent to a ðξv server where it can trigger an early warning system and, should the release tzar (that'd be taylor) deem it necessary, alert the core devs. through standardisation and usage of a simple format, it's designed so that we can easily code up a json-comparison engine to quickly diagnose where the consensus issue lies. one future job is to combine the two so that the json-rpc can also manage the reporting of canary information in this case, a user may set up a sentinel server to give them an email whenever the contract reports that mining has ceased and/or they should upgrade their node to continue mining. foundation finally, the three directors of the ethereum foundation (vitalik with three votes, then taylor and mihai with a vote each) will soon convene for, in vitalik's words, a "kind of great passing of the baton event". kelley, ðξv's indomitable coo, has been spearheading the search for proven candidates from a variety of fields who can add real value to the foundation both in terms of fundraising and of driving adoption. we're extremely grateful for all her hard work in this regard. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements introducing casper “the friendly ghost” | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search introducing casper “the friendly ghost” posted by vlad zamfir on august 1, 2015 research & development hi everyone – vlad here. i’ve been working on the analysis and specification of  “proof-of-stake” blockchain architecture since september 2014. while vitalik and i haven’t agreed on all of the details of the spec, we do have consensus on many properties of the proof-of-stake protocol that will likely be implemented for the serenity release! it is called casper “the friendly ghost” because it is an adaptation of some of the principles of the ghost (greedy heaviest-observed sub-tree) protocol for proof-of-work consensus to proof-of-stake. this blog post (my first one!) shares properties that are likely to be true of casper’s implementation in the serenity release. formal verification and simulation of casper’s properties is under way, and will be published eventually in the meantime, please enjoy this high-level, informal discussion!  : ) security-deposit based security and authentication casper is a security-deposit based economic consensus protocol. this means that nodes, so called “bonded validators”, have to place a security deposit (an action we call “bonding”) in order to serve the consensus by producing blocks. the protocol’s direct control of these security deposits is the primary way in which casper affects the incentives of validators. specifically, if a validator produces anything that casper considers “invalid”, their deposit are forfeited along with the privilege of participating in the consensus process. the use of security deposits addresses the “nothing at stake” problem; that behaving badly is not expensive. there is something at stake, and bonded validators who misbehave in an objectively verifiable manner will lose it. very notably, a validator’s signature is only economically meaningful so long as that validator currently has a deposit. this means that clients can only rely on signatures from validators that they know are currently bonded. therefore, when clients receive and authenticate the state of the consensus, their authentication chain ends in the list of currently-bonded validators. in proof-of-work consensus, on the other hand, the authentication chain ends in the genesis block as long as you know the genesis block you can authenticate the consensus. here, as long as you know the set of currently-bonded validators, you can authenticate the consensus. a client who does not know the list of currently bonded validators must authenticate this list out-of-band. this restriction on the way in which the consensus is authenticated solves the “long range attack” problem by requiring that everyone authenticate the consensus against current information. the validator list changes over time as validators place deposits, lose their deposits, unbond, and get unbonded. therefore, if clients are offline for too long, their validator list will no longer be current enough to authenticate the consensus. in the case that they are online sufficiently often to observe the validator set rotating, however, clients are able to securely update their validator list. even in this case, clients must begin with an up-to-date list of currently-bonded validators, and therefore they must authenticate this list out-of-band at least once. this “out-of-band authentication only necessarily once” property is what vitalik calls weak subjectivity. in this context information is said to be “objective” if it can be verified in a protocol-defined manner, while it is “subjective” if it must be authenticated via extra-protocol means. in weakly subjective consensus protocols, the fork-choice rule is stateful, and clients must initialize (and possibly sometimes renew) the information that their fork-choice rule uses to authenticate the consensus. in our case, this entails identifying the currently bonded validators (or, more probably a cryptographic hash of the validator list). gambling on consensus casper makes validators bet a large part of their security deposits on how the consensus process will turn out. moreover, the consensus process “turns out” in the manner in which they bet: validators are made to bet their deposits on how they expect everyone else to be betting their deposits. if they bet correctly, they earn their deposit back with transaction fees and possibly token issuance upon it – if on the other hand they do not quickly agree, they re-earn less of their deposit. therefore through iterated rounds of betting validator bets converge. moreover, if validators change their bets too dramatically, for example by voting with a high probability on one block after voting with a very high probability on another, then they are severely punished. this guarantees that validators bet with very high probabilities only when they are confident that the other validators will also produce high probability bets. through this mechanism we guarantee that their bets never converge to a second value after converging upon a first, as long as there there is sufficient validator participation. proof-of-work consensus is also a betting scheme: miners bet that their block will be part of the heaviest chain; if they eventually prove to be correct, they receive tokens whereas if they prove to be incorrect, they incur electricity costs without compensation. consensus is secured as long as all miners are betting their hashing power on the same chain, making it the blockchain with the most work (as a direct result of and as preempted by their coordinated betting). the economic cost of these proof-of-work bets add up linearly in the number of confirmations (generations of descendant blocks), while, in casper, validators can coordinate placing exponentially growing portions of their security deposits against blocks, thereby achieving maximum security very quickly. by-height consensus validators bet independently on blocks at every height (i.e. block number) by assigning it a probability and publishing it as a bet. through iterative betting, the validators elect exactly one block at every height, and this process determines the order in which transactions are executed. notably, if a validator ever places bets with probabilities summing to more than 100% at a time for a given height, or if any are less than 0%, or if they bet with more than 0% on an invalid block, then casper forfeits their security deposit. transaction finality when every member of a supermajority of bonded validators (a set of validators who meet a protocol-defined threshold somewhere between 67% and 90% of bonds) bets on a block with a very high (say, > 99.9%) probability, the fork-choice rule never accepts a fork where this block does not win, and we say that the block is final. additionally, when a client sees that every block lower than some height h is final, then the client will never choose a fork that has a different application state at height h 1 than the one that results from the execution of transactions in these finalized blocks. in this eventuality, we say that this state is finalized. there are therefore two relevant kinds of transaction finality: the finality of the fact that the transaction will be executed at a particular height (which is from finality of its block, and therefore priority over all future blocks at that height), and the finality of the consensus state after that transaction’s execution (which requires finality of its block and of unique blocks at all lower heights). censorship resistance one of the largest risks to consensus protocols is the formation of coalitions that aim to maximize the profits of their members at the expense of non-members. if casper’s validators’ revenues are to be made up primarily of transaction fees, for example, a majority coalition could censor the remaining nodes in order to earn an increased share of transaction fees. additionally, an attacker could bribe nodes to exclude transactions affecting particular addresses – and so long as a majority of nodes are rational, they can censor the blocks created by nodes who include these transactions. to resist attacks conducted by majority coalitions, casper regards the consensus process as a cooperative game and ensures that each node is most profitable if they are in a coalition made up of 100% of the consensus nodes (at least as long as they are incentivized primarily by in-protocol rewards). if p% of the validators are participating in the consensus game, then they earn f(p) ≤ p% of the revenues they would earn if 100% of the validators were participating, for some increasing function f. more specifically, casper punishes validators for not creating blocks in a protocol-prescribed order. the protocol is aware of deviations from this order, and withholds transaction fees and deposits from validators accordingly. additionally, the revenue made from betting correctly on blocks is linear (or superlinear) in the number of validators who are participating in at that height of the consensus game. will there be more transactions per second? most probably, yes, although this is due to the economics of casper rather than due to its blockchain architecture. however, casper’s blockchain does allow for faster block times than is possible with proof-of-work consensus. validators will likely be earning only transaction fees, so they have a direct incentive to increase the gas limit, if their validation server can handle the load. however, validators also have reduced returns from causing other, slower validators to fall out of sync, so they will allow the gas limit to rise only in a manner that is tolerable by the other validators. miners investing in hardware primarily purchase more mining rigs, while validators investing in hardware primarily upgrade their servers so they can process more transactions per second. miners also have an incentive to reinvest in more powerful transaction processing, but this incentive is much weaker than their incentive to purchase mining power. security-deposit-based proof-of-stake is very light-client friendly relative to proof-of-work. specifically, light clients do not need to download block headers to have full security in authenticating the consensus, or to have full economic assurances of valid transaction execution. this means that a lot of consensus overhead affects only the validators, but not the light clients, and it allows for lower latency without causing light clients to lose the ability to authenticate the consensus. recovery from netsplits casper is able to recover from network partitions because transactions in non-finalized blocks can be reverted. after a partition reconnects, casper executes transactions from blocks that received bets on the partition with higher validator participation. in this manner, nodes from either side of the partition agree on the state of the consensus after a reconnection and before validators are able to replace their bets. validator bets converge to finalize the blocks in the partition that had more validator participation, with very high probability. casper will very likely process the losing transactions from losing blocks after the ones from winning blocks, although it is still to be decided whether validators will have to include these transactions in new blocks, or if casper will execute them in their original order, himself. recovery from mass crash-failure casper is able to recover from the crash-failure of all but one node. bonded validators can always produce and place bets on blocks on their own, although they always make higher returns by coordinating on the production of blocks with a larger set of validators. in any case, a validator makes higher returns from producing blocks than from not producing blocks at all. additionally, bonded validators who appear to be offline for too long will be unbonded, and new bonders subsequently will be allowed to join the validation set. casper can thereby potentially recover precisely the security guarantees it had before the mass crash-failure. what is casper, in non-economic terms? casper is an eventually-consistent blockchain-based consensus protocol. it favours availability over consistency (see the cap theorem). it is always available, and consistent whenever possible. it is robust to unpredictable message delivery times because nodes come to consensus via re-organization of transactions, after delayed messages are eventually received. it has an eventual fault tolerance of 50%, in the sense that a fork created by >50% correct nodes scores higher than any fork created by the remaining potentially-faulty validators. notably, though, clients cannot be certain that any given fork created with 51% participation won’t be reverted because they cannot know whether some of these nodes are byzantine. clients therefore only consider a block as finalized if it has the participation of a supermajority of validators (or bonded stake). what is it like to be a bonded validator? as a bonded validator, you will need to securely sign blocks and place bets on the consensus process. if you have a very large deposit, you will probably have a handful of servers in a custom multisig arrangement for validation, to minimize the chance of your server misbehaving or being hacked. this will require experimentation and technical expertise. the validator should be kept online as reliably and as much as possible, for it to maximize its profitability (or for otherwise it will be unprofitable). it will be very advisable to buy ddos protection. additionally, your profitability will depend on the performance and availability of the other bonded validators. this means that there is risk that you cannot directly mitigate, yourself. you could lose money even if other nodes don’t perform well but you will lose more money yet if you don’t participate at all, after bonding. however, additional risk also often means higher average profitability especially if the risk is perceived but the costly event never occurs. what is it like to be an application or a user? applications and their users benefit a lot from the change from proof-of-work consensus to casper. lower latency significantly improves the user’s experience. in normal conditions transactions finalize very quickly. in the event of network partitions, on the other hand, transactions are still executed, but the fact that they can potentially still be reverted is reported clearly to the application and end-user. the application developer therefore still needs to deal with the possibility of forking, as they do in proof-of-work, but the consensus protocol itself provides them with a clear measure of what it would take for any given transaction to be reverted. when can we hear more? stay tuned! we’ll be sure to let you know more of casper’s specification over the next months, as we come to consensus on the protocol’s details. in addition, you can look forward to seeing simulations, informal and formal specification, formal verification, and implementations of casper! but please, be patient: r&d can take an unpredictable amount of time!  : ) previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements background on the mechanics of the ether pre-sale | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search background on the mechanics of the ether pre-sale posted by taylor gerring on july 9, 2014 organizational during the run-up to the ether sale we will be releasing a series of blog posts regarding our development plans and intended organizational structure. today we describe some of the background details of the ether sale module.  the sale module will be displayed on our website at https://www.ethereum.org for the duration of the sale. what is it? the ether sale is an html/javascript web application that enables people and organizations to pre-purchase ether tokens with bitcoin. ether may be purchased by parties interested in using it in their development of distributed applications on the ethereum platform or simply to pay for the use of applications on the platform once the genesis block is released.  the application runs mostly inside the browser, including pseudo random number generation (prng) and encryption/decryption of user secrets. additionally, we will be making available a standalone pure python application that can be used from a command line interface to purchase ether in lieu of using the website store. website workflow: after agreeing to the store’s terms and conditions, the buyer enters the desired amount of ether to purchase, their email address, a password to use for encryption, and generates a “random” seed forming the basis for their transaction-specific bitcoin & ethereum addresses. the user buyer sends payment in bitcoin to the generated address which is displayed on the store page and also provided in a downloadable encrypted wallet file. when the browser, which is monitoring the bitcoin blockchain, sees enough funds have been sent to this generated address, the browser creates a new transaction forwarding the bitcoin to the genesis sale address. this transaction is signed by the buyer’s browser and submitted to the bitcoin network. no transaction data is stored on the any servers because the bitcoin blockchain serves as the purchase database. after the transaction has been successfully submitted, the user buyer is again prompted to download a backup copy of their encrypted wallet in addition to being emailed another backup file.  the buyer should be aware that keeping the wallet as an email attachment in an email account for any period of time represents a vulnerability.  the email attachment should be downloaded to a safe computer at the earliest opportunity and the email should be expunged from the email system. (what follows is technical information, the gui we have developed abstracts all of this is in a very user-friendly manner) wallet encryption process:  randomgen a seed. bkpkey = sha3(seed + “\x02″)[:16] key = pbkdf2(pw) encseed = encrypt(key, seed) wallet = [ encseed, bkpkey ] (plus non-sensitive data like ethereum and bitcoin address) the btc and eth privkeys and addresses are derived from the seed. seed recovery process:  wallet + pw recovery:  (normal) key = pbkdf2(pw) seed = decrypt(key, encseed) overview of website steps for making an ether purchase: buyer clicks “buy ether” and agrees to the terms & conditions buyer enters initial purchase information buyer enters the amount of ether they’d like to purchase buyer enters email address as the destination for wallet backup file buyer enters passphrase, which serves as an encryption seed. a very strong passphrase is highly recommended buyer generates entropy by moving their mouse or tapping their screen. a “seed” is constructed based on this generated entropy as well as other random system inputs. when the desired entropy length is achieved the wallet is generated using the seed.  the wallet contains: an individual btc address to send the funds to an individual eth address an encrypted seed a backup encrypted seed the browser sale app checks the newly created btc address to see if funds arrive. before sending any funds, the buyer has an opportunity to download the wallet. if a problem occurs between transactions, any funds send to the buyer’s personal deposit address will be accessible with the passphrase. if the unspent balance of the btc address > 0.01 btc, the browser sale app generates a signed transaction from the newly created btc address with 2 specific outputs: an output of the total unspent balance minus miners fee to the main sale btc address–this btc address is where all the funds go, it’s a fixed, known btc address. an output of 10000 satoshi to a btc address generated from the (string) eth address–this bitcoin address is for confirming ethereum addresses and is unique for every transaction. the browser sale app prompts the purchaser to download their wallet again & and also sends the buyer an email containing a backup for the ether wallet.  the same caution as stated above applies.  do not leave the wallet as an email attachment in an email system.  download as soon as possible and ensure that the email is expunged. the browser sale app will display the number of bitcoin transaction confirmations so that’s it! although there’s a lot of technical things happening in the background, the gui we have developed will make the process a click by click operation. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the ethereum project: learning to dream with open minds | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the ethereum project: learning to dream with open minds posted by mihai alisie on july 14, 2014 research & development “and those who were seen dancing were thought to be insane by those who could not hear the music.” friedrich nietzsche ethereum as a project is more than just a technology, even if technology is at its core. in the rush of creating, co-creating and debugging everything you sometimes lose sight of the bigger picture. you need stop for a second, take a step back and observe in order to fully appreciate what's going on. the project started as an idea in vitalik’s mind. ethereum, has now expanded into thousands and thousands of other minds around the world, thanks to this interconnected digital mesh called internet. a sea of minds brought together by the idea of a turing complete general purpose blockchain is creating, as you’re reading these lines, the minecraft of blockchain tech. in all this excitement we have had to think also, about how we can best ensure that the vision is not corrupted, while delivering the technology itself. this post is the first in a series that attempts to explain both the ethos and motivation behind the project as a whole. soulsearching: for profit or not? from inception, our intention was to deliver a technology aligned with the community at large and to encourage a meritocratic culture based on trust, openness and respect not only for our fellow ethereans but everyone united by raw passion for positive world change. over the past few months the model of our organization and the very nature of this project has been in discussion both amongst ourselves and between ourselves and the community. they were healthy discussions and i would like to personally thank everyone who cared enough to share their thoughts on our forums, in a blog comment or a reddit post. it has been an intense and interesting journey. along the way it made us reflect upon our values while looking at ourselves and asking what this project really means to each and all of us. during these long soul searching sessions we realized that most of the dichotomies boiled down to a single question: for profit or not? after researching a number of models and organizations, we arrived at the conclusion that in order to keep the protocol and the software as pure as possible the only real option was to build a non-profit structure with a declared mission to develop, maintain, nurture and explore the potential of this new technology. motivated not by money but radiant passion for this crazy idea of a free, open and decentralized world. that’s what make us happy in the end and most importantly what makes us get up and work on this project, proudly saying today we're building tomorrow. details regarding the non profit structure chosen will follow in a series of blogs. however this blog post is not about details, more about the bigger picture and why we’re doing it in the first place. continuing the experiment we all come from different backgrounds and life experiences, but in the end we're all connected through this meaningful project which enables us to explore our passions and ideas through code, art, hardware or sometimes something completely new. by design this project is breaking new ground in a way that has not been tried before. this by definition is a trial and error experiment, looking to continue the experiment started by bitcoin, on multiple levels. since this has never been tried before, there are no right or wrong answers or concrete experiences to fall back on. we are deep into uncharted territories. even if we do not yet grasp the full applications and implications, the sheer number of ideas and initiatives sparked by the core idea itself is encouraging, and gives us a glimpse at the limitless potential of our minds we are architecting and co-creating our own future "reality". i like to look at the ethereum project as a blank canvas, where anyone can create digital masterpieces. the technology behind it serving as a catalyst, enabling people to unfold their innate playful creativity. if the network and blockchain were to be the canvas, then the paints and colors would be the lines of code running transparently and censorlessly on top. everyone is invited to cocreate. the genesis sale represents the next step in this epic journey that lies ahead, and a way to bootstrap the ethereum ecosystem. the collective intelligence of the swarm will then have its collective fingertips at an amazing open technology, capable of unleashing waves of innovation across the web, and etherweb. with a technology like this available, centralized, corrupted monopolies that siphon from the blood and tears of humanity face rapid obsoletion. portal into greatness "don't ask yourself what the world needs. ask yourself what makes you come alive and then go do that. because what the world needs is people who have come alive." howard thurman most of us are aware that the current planetary operating system is pushing us into extinction as a species. finance seems to be the current paradigm humanity has to transcend on its path from a civilization type zero into a civilization type one. when a technology like ethereum exists, i feel like we almost have a debt to ourselves and our fellow species to explore its potential in the search of solutions. we truly have at our fingertips the power to craft the future we desire through code and purposeful action if you see something you don’t like, contribute with something you do like. we believe it is important, especially at this critical point in time, to follow our heart instead of meaningless extrinsic rewards. this is how, perhaps, we can open a portal toward our maximum potential, both on an individual and collective level. i suspect that sir timothy berners-lee was most likely unable to envision something like bitcoin running on top of his hyper text unified network protocol back in early 90’s. just as we are unable at this point to fully grasp the implications of the ongoing crypto renaissance running across the ubiquitous digital mesh that connects more than 2 billion minds. how far can we take it? what's the limit? is there really a limit in the first place? from our humble beginnings as foragers and hunters, we have arrived at a point where we are able to contemplate the nature of infinity. able to grasp the idea that we’re made from trillions of atoms vibrating in unison on a pale blue dot, all this while permanently connected through a digital planetary consciousness grid. we are learning to dream with open minds, just as we are dreaming with closed eyes. we are on the cusp of changing the world with our minds through lines of code dancing with atoms at the edge of singularity. the interesting part is just about to begin. how many rivers will we have to cross before we'll find our way? previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the xbox and ethereum’s dual mandate | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the xbox and ethereum’s dual mandate posted by joseph lubin on may 15, 2014 research & development the necessity to build and launch an ecosystem on genesis day imagine if microsoft released the xbox and there were no games. on the day on which the ethereum genesis block is created, many of the elements of the ecosystem — core infrastructure, mining network, user app browser and applications — will be in place.  granted there will be an enormous amount of development to do beyond the genesis to craft ethereum into a sophisticated, decentralized consensus (decentcon) application platform.  but on day one, there will be “launch titles.” originally, when the idea behind ethereum was conceived in november, the intent was for it to be a simple “altcoin” (in fact, a meta-protocol on top of primecoin) with a scripting language built in. over time, however, as we realized the project’s potential our scope necessarily grew, and our current intent is to release a flagship product called the “etherbrowser:” a browser that can browse both the old-school internet and the decentralized web (web3.0).  from the moment of inception of the ethereum network, the following elements in the etherbrowser and the ecosystem will be operational: a fully functional implementation of the ethereum blockchain which enhances the bitcoin technology with one-minute block times and stronger security (ghost protocol), which decentralized applications (“ðapps”) can use for anything that requires consensus (eg. currency and token systems, name registries, proof of existence, etc., …) a fully functional peer-to-peer protocol, which ðapps can use to send messages between users. for example, a decentralized messaging ðapp will use this protocol for actual message sending, but the blockchain for registering user accounts. an elegant, browser-and-app-store user interface for using ðapps network nodes (miners) that process transactions and computations, secure the network through a consensus process and are compensated by fees and the issuance of new ether a standard set of “built-in” utility-level ðapps: distributed wallet, distributed id system, distributed reputation system, distributed messaging system, distributed name registry service, distributed catalog of distributed applications, etc., will serve as just a few of the low-level utilities on the system a standard set of built-in tools for both developer and end-user: blockchain explorer, contract explorer, transaction constructor, encrypted messaging tool, signature tool, authentication tool, ….  and a debugger will be available for the hard core. and soon after the genesis we hope to see a rich set of third-party distributed applications, e.g. distributed storage services, distributed notary services, distributed exchange services, distributed escrow and arbitration services, financial contracts, insurance contracts, lending services, purposed reputation systems, games, entertainment services, etc., …  each of the ðapps in the different business verticals will be able to make use of the lower level, system utility ðapps like the id and reputation systems. without operational distributed applications, ethereum is a hollow shell.  in order to have a broad offering of distributed applications from day one, non-profit, public-good elements must be built in conjunction with commercially driven applications in competitive markets.  from this requirement, ethereum takes its dual mandate. it can be argued that certain elements within an ecosystem should be developed for the common good, and do not benefit from a profit motive.  certain other elements, are better developed in a competitive, for-profit context.  for instance, it makes some sense to develop a coherent, large-scale, systematized road system for the public good, using public tax dollars.  but it probably does not make sense to build cars with public tax dollars, since development under the pressures of free market competition forces the design and production of better products and benefits the market as a whole. dual mandate: collaborative non-profit and for-profit development two hub organizations will lead the ethereum project under dual mandate: a for-profit entity based in zug, switzerland, and a non-profit organization, based in toronto, canada.  both of these organizations will be constructed in a fashion that will enable each to remain strong and financially independent to ensure years of development and growth of the ecosystem. the ethereum ecosystem will initially be constructed by the non-profit and for-profit entities in partnership, ideally with broad participation from many other independent entities.  the founding leadership will put into place mechanism that ensures that the foundational elements of the ecosystem are built and maintained in a fashion that adheres to the founding principles.  funding and independence will be guaranteed to the organizations (either state-registered businesses or eventually blockchain-registered businesses) that guide the various avenues of development of the core infrastructure. at the same time, human economies are largely about free market commerce (or should be), so the non-profit organization, to some extent — and especially the for-profit entity — will assist independent business efforts in the space to launch and flourish.  this assistance can take a variety of forms from providing resources and guidance, to maintaining support channels, to the creation of joint ventures. the hybrid organization model “mozilla is one such [hybrid non-profit/for-profit] organization and, as i look around, i see others. there is alot [sic] of upside to how these orgs work, especially the potential to move markets towards the public good at a global scale. “…the more i dig, the more i am convinced that organizations like mozilla, wikipedia, kiva, miro and so on that mashup mission, market and the culture of the web represent a new pattern worth understanding. “mission + market + web hybrids matter because they can wield the power needed to move markets at a global scale, while still looking out for the small guy, taking the long view and staying true to their public benefit mission. they show us how organizations could — and maybe should — work in the future.” mark surman, executive director, mozilla foundation trolltech/digia (qt), wordpress, red hat, google android and the open handset alliance, the mozilla organization and several others have pioneered the dual mandate approach and built thriving ecosystems and successful companies.  the model that sun developed in the late 90s is also instructive. sun microsystems (now oracle) developed the java computer language and libraries in house. as innovative as it was, it is likely that it would not have acquired such large market share, had sun not realized that they, as a for-profit company, could not control a foundational aspect of software development that it, and many of its competitors, were expected to use.  to address this conflict of interest, sun open sourced java and developed the java community process to shepherd growth of the platform in a nonpartisan fashion. ethereum is an open source project and will remain open source.  and the ethereum project, under the auspices of its non-profit foundation will create a community development process philosophically similar to the java community process.  in fact, ethereum, thus far has been developed under a less formalized community development process. the ethereum non-profit entity will specify and guide the development of the core ethereum infrastructure.  it will employ a non-partisan community development process and will develop, conduct and support research initiatives in the general cryptocurrency space.  the non-profit will have a membership composed originally of the ethereum founders, and soon will add other independent leaders in the emerging space. the ethereum for-profit entity will be charged with the task of developing certain elements of the core infrastructure in collaboration with the non-profit entity and building out the ecosystem with distributed application services and businesses that it develops itself, funds, or otherwise supports. we believe this to be the most effective structure in the near term.  that said, we are daoists (where dao means distributed autonomous organization).  we will look to move as much infrastructure and governance as soon as possible onto the blockchain.  our opensalary system will probably be the first component to move to the blockchain. please stay tuned for more on the topics of openorganization and daoification of the two organizations.  our goal, if we do our jobs properly, is to put ourselves out of (state-registered) business. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements gav’s ðξv update i: where ethereum’s at | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search gav’s ðξv update i: where ethereum’s at posted by gavin wood on october 17, 2014 research & development who are you? i'm gav together with jeffrey wilcke and vitalik buterin, i'm one third of the ultimate leadership of ethereum ðξv. ðξv is a uk software firm that is under a non-profit-making agreement with the ethereum foundation to create version 1.0 of the web three software stack. we three directors---who are ultimately responsible that the software is built and works---are the same three developers who designed and implemented the first working versions of the ethereum clients. ðξv is geographically split between london (where our comms operation is based) and berlin (which hosts the main hub of ðξv). though i'm based in zug, switzerland (being an ethereum employee), i have been involved most recently in putting together the berlin side of things. since its inception in summer, we have been working to set up the technical side of the project, under which we include our communications, education and adoption team lead by stephan tual and helped by mathias grønnebæk for the organisation of operations. a berlin who's who aeron buchanan, though originally brought on as a mathematical modeller, has been very successful in coordinating berlin's various operations including helping set up the arduous process of getting a bank account, recruitment, financial juggling to get people paid, technical interviews and other tedious administration tasks; more recently he has also been helping sort out the uk side of things, too. i must acknowledge brian fabian who helped connect us while in berlin and made it possible for us to have a legal structure in place quickly. at present, the operation in berlin is directed by our major pyethereum contributor, heiko hees, with aeron being the essential point of control for all operations. over time, we expect aeron to get back to modelling and to find a suitable candidate for the day-to-day management of the hub. during our time in berlin we've been very active in hiring (which as a process is considerably more arduous than you might think): alex leverington was our first hire and he flew to berlin all the way from texas to join the team. alex has been engaged helping out with the mac builds and making volunteer contributions since early in the year, so it's great that he wanted to step forward into a permanent role. now alex has been working on some of the internals of the c++ client (specifically the client multiplexing, allowing multiple web three applications to coexist on the same physical machine). over the past few months we've recruited a few more people: one of our first hires following alex is christian, who holds a phd in multiobjective optimization and language equations and is now engaged in prototyping and implementing the new domain-specific contract-authoring language that i proposed a while ago, solidity. at around the same time, we also hired christoph, currently finishing his phd in physics and who utterly loves writing unit tests. it didn't take me long to realise he'd be great for sorting out our clients' interoperability issues. he has been leading our recent surge in getting the protocol in alignment for all clients through a comprehensive code-covering set of unit tests for the virtual machine operation. our newest recruit, marek, journeying to berlin from poland, was an early ethereum volunteer and enjoyed making contracts on some early c++ client prototypes. coming from a web-technology background (though being perfectly competent in c++), he has now started working on our c++/javascript api, aiming towards full node.js integration to facilitate backend integration with existing web sites. working alongside marek is marian, the feathers in whose cap include much of the technical prowess behind the highly successful ether sale. he has been developing out some of the first web apps to use ethereum as its backend. i look forward to welcoming three more hires in the coming weeks, including some personnel with rather impressive and uniquely relevant backgrounds. more news on that next time. and more... back in london, we've hired design outfit proof-of-work, headed by louis chang, to put together our new website and brand. we're ecstatic with how things are coming along there and look forward to unveiling it soon. once this is in place we'll have a much clearer way of getting our updates and information out regarding what's happening at ðξv. externally to ðξv but supported by it are a number of other individuals and projects: i am very grateful to tim, who continues to consult on our efforts at an asic-resistant proof-of-work algorithm, also implementing it in c++. similarly, caktux an early volunteer and maintainer of the ncurses-based c++ ethereum front-end neth has been invaluble (alongside joris and nick savers) in getting a continuous integration system up and running. we are pleased to support both of them in their endeavours to make this project a success. furthermore the guys at imapp, a software firm in warsaw specialising in advanced languages and compilers deserve a great nod for their on-going efforts at using their considerable expertise in implementing a just-in-time (jit) compiled version of the ethereum virtual machine and making computationally-complex contracts a reasonably affordable possibility. finally, i must thank the ethereumj (java client) volunteer developers roman and nick, both of whom have visited us in our prototype hub here in berlin, and who work tirelessly to find different and innovative new ways of interpreting the formal protocol specification. the california connection over in silicon valley, we have made two hires, joseph and martin; joseph will be leading the efforts there and concentrating on developing some of our core ðapps that will help demonstrate the potential of ethereum. martin is leading the effort to create a pure javascript implementation of ethereum, a lofty goal, and thus all the more impressive that the project now has a core that is compatible with poc-6. we are also looking forward to working with the agreemint foundation (ie. mintchalk), with their effort to create an online contract development environment, to provide a simple and highly accessible interface for the beginner and intermediate level users to learn about contract development and create and deploy web three ðapps. in the future we hope to expand our operations there, particularly over january and february when vitalik and i will be staying there, we in particular look forward to spending some time discussing the future of data sharing and online publication with juan of ipfs and are optimistic about the possibility of finding some synergy between our projects. on go-ing development though i'm sure jeff will make his own post on the goings-on over at his golang-orientated end, i will say that on a personal note i'm very happy that alex (aka avsa) has joined us on a permanent basis. alex is well known on the ethereum forums and his mockups of what web three could look like were simply incredible in insight, technical knowledge and polish. as an accomplished ui & ux engineer, he'll be joining jeff in taking mist, the web three browser, forward and making it into what i am sure will simultaneously be the most revolutionary and pleasing to use piece of new software in a very long time. so what's happening in berlin then? when we arrived at first we needed somewhere to be based out of: thanks to brian, we were invited to the rainmaking loft, an excellent space for tech startups that need somewhere to spread their rug prior to world domination. since august we've had a nice big desk there for our developers to work alongside our inimitable location scout, hub outfitter, project manager and interior designer rolled into one; sarah. sarah has worked tirelessly in finding our perfect location, our perfect contractors and our perfect fixtures and fittings and making it actually work. right now as i write this at 4am eest, she's probably up on ebay looking for a decent deal for office chairs or costing a well-placed dry wall. and what a job she has done thus far. we will be based in probably the most perfect place we could hope for. walking distance to two u-bahn stations, we're located on a quiet street adjacent to oranienstraße and a central point of kreuzberg. we're a short cycle ride from the centre of berlin's mass and, in the opposite direction, from the beautiful canal and neukölln. we have some lovely quiet bars and cafés on our sexy little street and the bustling new-tech area that is kreuzberg at the end of it. our new hub, designed and outfitted by her will be a 250m² cross of office, homely relaxation environment and (self-service) café---a new (and german-building-law-friendly) twist on the notion of the holon. we'll be able to host meetups and events, have a great area for working and have ample collaboration space for any other ethereum-aligned operations that would prefer not to pay coffee-tax for their power & wifi. not to be forgotten, helping aeron and me with administration, procurements and organisation, not to mention general german-speaking tasks, lisa has been invaluable during this period. and what have we been doing? since beginning, ðξv's time has inevitably been wast^h^h^h^h spent wisely in bureaucracy, administration and red tape. it is impressive how much of a pain doing business in a perfectly well developed nation like germany can be. slowly (and thanks in no small part to aeron) this tediousness is starting to let up. when not engaged in such matters, we've been pressing to get our most recent proof-of-concept releases out, poc-5 and poc-6. poc-5 brought with it a number of important alterations to the ethereum virtual machine and the core protocol. poc-6 brought a 4-second block time (this is just for stress-testing; for the mainet we're aiming for a 10 second block time) and wonderfully fast parallel block-chain downloading. furthermore we've been talking with various potential technology partners concerning the future of swarm, our data distribution system, including with our good friend juan batiz-benet (vitalik & i got to know him while staying at his house in silicon valley for a week back in march). speaking at a few meetings and conferences has taken time also. in my case, the keynote speaker at both inside bitcoin and latin america's popular tech-fest campus party was an honour, as was the invitation to address the main hall at the wonderful university san francisco of quito. i hesitate to imagine the number of such engagements vitalik has done during the same time period. in addition to his impressive public speaking schedule, vitalik has been putting in considerable efforts into research on potential consensus algorithms. together with vlad, a number of potential approaches have been mooted over the past few weeks. ultimately, we decided to follow the advice of some in our community, like nick szabo, who have urged us to focus on getting a working product off the ground and not try to make every last detail perfect before launching. in that regard, we've decided to move many of our more ambitious changes, including native extensions, auto-triggering events and proof of stake, into a planned future upgrade to happen around mid-to-late 2015. however, during a two-week visit to london vitalik made major progress working with vlad on developing stable proof-of-stake consensus algorithms, and we have a few models that we think are likely to work and solve all of the problems inherent in current approaches. the two have also begun more thoroughly laying the plans for our upcoming upgrades in scalability. more recently, i have been hard at work rewriting much of the networking code and altering the network protocol to truly split off the peer-to-peer portion of the code to make an abstract layer for all peer-to-peer applications, including those external to the web three project that wish to piggyback on the ethereum peer network. i've also been getting poc-7 up to scratch and more reliable, as well as upgrading my team's development processes which predictably were becoming a little too informal for an increasingly large team. we'll be moving towards a peer-reviewed (rather than gav-reviewed) commit review process, we have a much more curated github issue tracker, alongside an increasingly scrum-oriented project management framework (a switch to pivotal tracker is underway everything public, of course). most recently i've been working on the whisper project, designing, developing, chewing things over and prototyping. finally, we've also been making inroads into some well-known and some other not-so-well-known firms that can help us make our final core software as safe and secure as humanly possible. i'm sorry i can't go into anything more specific now, but rest assured, this is one of our priorities. so there you have it. what's been happening. and what's going to happen? aside from the continuing hiring process and our inroads into setting up a solid security audit, we will very soon be instituting a more informal manner for volunteers and contributors to be supported by the project. in the coming days we will be launching a number of ðξv schemes to make it possible for dedicated and productive members of the ethereum and web three community to apply for bursaries and expenses for visiting us at one of our hub locations. watch this space. in terms of coding, ðξv, at present, has one mission: the completion of version 1.0 of the ethereum client software which will enable the release of the genesis block. this will be done as soon as possible, though we will release the genesis block only when we (and many others in the security world) are happy that it is safe to do so: we are presently aiming to have it out sometime during this winter (i.e. between december 21st and march 21st). this will include at least a basic contract development environment (the focus of the work here in berlin under myself), an advanced client based around google's chromium browser technology and several core ðapps (the focus of the work under jeff), and various command-line tools. in specifics, after we have poc-7 out, we'll be making at most one more proof-of-concept release before freezing the protocol and moving into our alpha release series. the first alpha will signal the end of our core refactoring & optimisation process and the beginning of our security audit; we aim to have this under way within the next 4-6 weeks. the security audit will involve a number of people and firms, both internal and external, both hired and incentivised, analysing the design and implementations looking for flaws, bugs and potential attack vectors. once all parties involved have signed off on all aspects of the system will we move to organise a coordinated release of the final block chain. we expect the auditing process to take 2-3 months, with another couple of weeks to coordinate the final release. during this process we will be developing out the other parts of the project, including the whisper messaging protocol, the contract development environment and solidity, the ethereum browser, mist and the core ðapps, all in readiness for the genesis block release. we will take a very much fluid attitude to software development & release and incrementally roll out updates and improvements to our core suite of software over time. we don't want to keep you waiting with the release of the blockchain and so that is our development priority. so you may be assured, it will be released just as soon as it is ready. so hold on to your hats! you'll be coding contracts and hacking society into new forms before you know it. gav. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the issuance model in ethereum | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the issuance model in ethereum posted by joseph lubin on april 10, 2014 research & development ether (eth), the cryptofuel that powers distributed applications on the ethereum platform, will be issued at a constant annual linear rate via the block mining process. this rate is 0.3 times the total amount of eth that will be purchased in the pre-sale. while the best metaphor for eth is “fuel for running the contract processing engine,” for the purposes of this post, we will treat eth purely as a currency. there are two common definitions of “inflation.”  the first relates to prices and the second relates to the total amount of money in a system – the monetary base or supply.  similarly for the term “deflation.”  in this post we will distinguish between “price inflation,” the rise in the general price level of goods and services in an economy, and “monetary inflation,” the growth in the supply of money in an economy due to some sort of issuance mechanism.  often, but not always, monetary inflation is a cause of price inflation. though the issuance of eth is in a fixed amount each year, the rate of growth of the monetary base (monetary inflation) is not constant.  this monetary inflation rate decreases every year making eth a disinflationary currency (in terms of monetary base).  disinflation is a special case of inflation in which the amount of inflation shrinks over time. it is expected that the amount of eth that will be lost each year caused by transmissions to addresses which are no longer accessible is estimated to be on the order of 1% of the monetary base. eth may be lost due to loss of private keys, death of owner without transmission of private keys, or purposeful destruction by sending to an address that never had an associated private key generated. if we assume that ethereum sells 40,000 btc worth of eth in the pre-sale, and if we assume that the average price is 1500 eth/ btc, 60,000,000 eth will be created in the genesis block and assigned to purchasers. every year, in perpetuity, 18,000,000 eth will be issued though the mining process.  taking into account both creation of new eth and loss of existing eth, in the first year, this represents a monetary inflation rate of 22.4%.  in the second year the rate drops to 18.1%.  by the tenth year, the rate is 7.0%.  in year 38, it hits 1.9%. and in the 64th year, the level of 1.0% is reached. figure 1. amount of eth in existence (dark green curve) on the left axis. monetary base inflation rate (light green curve) on the right axis. years on the horizontal axis. (adapted from arun mittal with thanks.) by approximately the year 2140, the issuance of btc ceases and since some btc will likely be lost each year, the monetary base of bitcoin is expected to start shrinking at that point. at approximately the same time, the expected rate of annual loss and destruction of eth will balance the rate of issuance.  under this dynamic, a quasi-steady state is reached and the amount of extant eth no longer grows. if the demand for eth is still growing at that point due to an expanding economy, prices will be in a deflationary regime.  this is not an existential problem for the system since eth is theoretically infinitely divisible. as long as the rate of price deflation is not too rapid, pricing mechanisms will adjust and the system will operate smoothly.  the traditional main objection to deflationary economies, wage stickiness, is likely not to be an issue since all payments systems will be fluid.  another frequent objection, borrowers forced to repay loans with a currency that grows in purchasing power over time, will also not be a problem if this regime is persistent, since terms of lending will be defined to account for this. note that while the monetary inflation remains greater than zero for many years, price levels (tracked as price inflation and deflation) are dependent on supply and demand, so are related to, but not totally controlled by the rate of issuance (supply).  over time it is anticipated that growth of the ethereum economy will significantly outpace growth of the supply of eth, which could lead to an increase in the value of eth with respect to legacy currencies and btc. one of bitcoin’s great value propositions was the algorithmically fixed total issuance of the currency which mandated that only 21,000,000 btc will ever be created.  in a time of profligate legacy currency printing in an exponentially doomed attempt to patch over the fact that there is too much debt in the global economic system (with more debt), the prospect of a universally accepted cryptocurrency that can serve eventually as a relatively stable store of value is attractive.  ethereum recognizes this and seeks to emulate this core value proposition. ethereum also recognizes that a system intended to serve as a distributed, consensus-based application platform for global economic and social systems, must strongly emphasize inclusiveness. one of the many ways we intend to foster inclusiveness is by maintaining an issuance system which possesses some churn.  new participants in the system will be able to purchase new eth or mine for new eth whether they are living in the year 2015 or 2115. we believe we have a achieved a good balance between the two goals of fostering inclusiveness and maintaining a stable store of value. and the constant issuance, especially in the early years, will likely make using eth to build businesses in the ethereum economy more lucrative than hoarding speculatively. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements an information-theoretic account of secure brainwallets | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search an information-theoretic account of secure brainwallets posted by vitalik buterin on october 23, 2014 research & development an important and controversial topic in the area of personal wallet security is the concept of "brainwallets" storing funds using a private key generated from a password memorized entirely in one's head. theoretically, brainwallets have the potential to provide almost utopian guarantee of security for long-term savings: for as long as they are kept unused, they are not vulnerable to physical theft or hacks of any kind, and there is no way to even prove that you still remember the wallet; they are as safe as your very own human mind. at the same time, however, many have argued against the use of brainwallets, claiming that the human mind is fragile and not well designed for producing, or remembering, long and fragile cryptographic secrets, and so they are too dangerous to work in reality. which side is right? is our memory sufficiently robust to protect our private keys, is it too weak, or is perhaps a third and more interesting possibility actually the case: that it all depends on how the brainwallets are produced? entropy if the challenge at hand is to create a brainwallet that is simultaneously memorable and secure, then there are two variables that we need to worry about: how much information we have to remember, and how long the password takes for an attacker to crack. as it turns out, the challenge in the problem lies in the fact that the two variables are very highly correlated; in fact, absent a few certain specific kinds of special tricks and assuming an attacker running an optimal algorithm, they are precisely equivalent (or rather, one is precisely exponential in the other). however, to start off we can tackle the two sides of the problem separately. a common measure that computer scientists, cryptogaphers and mathematicians use to measure "how much information" a piece of data contains is "entropy". loosely defined, entropy is defined as the logarithm of the number of possible messages that are of the same "form" as a given message. for example, consider the number 57035. 57035 seems to be in the category of five-digit numbers, of which there are 100000. hence, the number contains about 16.6 bits of entropy, as 216.6 ~= 100000. the number 61724671282457125412459172541251277 is 35 digits long, and log(1035) ~= 116.3, so it has 116.3 bits of entropy. a random string of ones and zeroes n bits long will contain exactly n bits of entropy. thus, longer strings have more entropy, and strings that have more symbols to choose from have more entropy. on the other hand, the number 11111111111111111111111111234567890 has much less than 116.3 bits of entropy; although it has 35 digits, the number is not of the category of 35-digit numbers, it is in the category of 35-digit numbers with a very high level of structure; a complete list of numbers with at least that level of structure might be at most a few billion entries long, giving it perhaps only 30 bits of entropy. information theory has a number of more formal definitions that try to grasp this intuitive concept. a particularly popular one is the idea of kolmogorov complexity; the kolmogorov complexity of a string is basically the length of the shortest computer program that will print that value. in python, the above string is also expressible as '1'*26+'234567890' an 18-character string, while 61724671282457125412459172541251277 takes 37 characters (the actual digits plus quotes). this gives us a more formal understanding of the idea of "category of strings with high structure" those strings are simply the set of strings that take a small amount of data to express. note that there are other compression strategies we can use; for example, unbalanced strings like 1112111111112211111111111111111112111 can be cut by at least half by creating special symbols that represent multiple 1s in sequence. huffman coding is an example of an information-theoretically optimal algorithm for creating such transformations. finally, note that entropy is context-dependent. the string "the quick brown fox jumped over the lazy dog" may have over 100 bytes of entropy as a simple huffman-coded sequence of characters, but because we know english, and because so many thousands of information theory articles and papers have already used that exact phrase, the actual entropy is perhaps around 25 bytes i might refer to it as "fox dog phrase" and using google you can figure out what it is. so what is the point of entropy? essentially, entropy is how much information you have to memorize. the more entropy it has, the harder to memorize it is. thus, at first glance it seems that you want passwords that are as low-entropy as possible, while at the same time being hard to crack. however, as we will see below this way of thinking is rather dangerous. strength now, let us get to the next point, password security against attackers. the security of a password is best measured by the expected number of computational steps that it would take for an attacker to guess your password. for randomly generated passwords, the simplest algorithm to use is brute force: try all possible one-character passwords, then all two-character passwords, and so forth. given an alphabet of n characters and a password of length k, such an algorithm would crack the password in roughly nk time. hence, the more characters you use, the better, and the longer your password is, the better. there is one approach that tries to elegantly combine these two strategies without being too hard to memorize: steve gibson's haystack passwords. as steve gibson explains: which of the following two passwords is stronger, more secure, and more difficult to crack? d0g..................... prxyc.n(n4k77#l!evdafp9 you probably know this is a trick question, but the answer is: despite the fact that the first password is hugely easier to use and more memorable, it is also the stronger of the two! in fact, since it is one character longer and contains uppercase, lowercase, a number and special characters, that first password would take an attacker approximately 95 times longer to find by searching than the second impossible-to-remember-or-type password! steve then goes on to write: "virtually everyone has always believed or been told that passwords derived their strength from having “high entropy”. but as we see now, when the only available attack is guessing, that long-standing common wisdom . . . is . . . not . . . correct!" however, as seductive as such a loophole is, unfortunately in this regard he is dead wrong. the reason is that it relies on specific properties of attacks that are commonly in use, and if it becomes widely used attacks could easily emerge that are specialized against it. in fact, there is a generalized attack that, given enough leaked password samples, can automatically update itself to handle almost anything: markov chain samplers. the way the algorithm works is as follows. suppose that the alphabet that you have consists only of the characters 0 and 1, and you know from sampling that a 0 is followed by a 1 65% of the time and a 0 35% of the time, and a 1 is followed by a 0 20% of the time and a 1 80% of the time. to randomly sample the set, we create a finite state machine containing these probabilities, and simply run it over and over again in a loop. here's the python code: import random i = 0 while 1: if i == 0: i = 0 if random.randrange(100) < 35 else 1 elif i == 1: i = 0 if random.randrange(100) < 20 else 1 print i we take the output, break it up into pieces, and there we have a way of generating passwords that have the same pattern as passwords that people actually use. we can generalize this past two characters to a complete alphabet, and we can even have the state keep track not just of the last character but the last two, or three or more. so if everyone starts making passwords like "d0g.....................", then after seeing a few thousand examples the markov chain will "learn" that people often make long strings of periods, and if it spits out a period it will often get itself temporarily stuck in a loop of printing out more periods for a few steps probabilistically replicating people's behavior. the one part that was left out is how to terminate the loop; as given, the code simply gives an infinite string of zeroes and ones. we could introduce a pseudo-symbol into our alphabet to represent the end of a string, and incorporate the observed rate of occurrences of that symbol into our markov chain probabilities, but that's not optimal for this use case because far more passwords are short than long, it would usually output passwords that are very short, and so it would repeat the short passwords millions of times before trying most of the long ones. thus we might want to artificially cut it off at some length, and increase that length over time, although more advanced strategies also exist like running a simultaneous markov chain backwards. this general category of method is usually called a "language model" a probability distribution over sequences of characters or words which can be as simple and rough or as complex and intricate as needed, and which can then be sampled. the fundamental reason why the gibson strategy fails, and why no other strategy of that kind can possibly work, is that in the definitions of entropy and strength there is an interesting equivalence: entropy is the logarithm of the number of possibilities, but strength is the number of possibilities in short, memorizability and attackability are invariably exactly the same! this applies regardless of whether you are randomly selecting characters from an alphabet, words from a dictionary, characters from a biased alphabet (eg. "1" 80% of the time and "0" 20% of the time, or strings that follow a particular pattern). thus, it seems that the quest for a secure and memorizable password is hopeless... easing memory, hardening attacks ... or not. although the basic idea that entropy that needs to be memorized and the space that an attacker needs to burn through are exactly the same is mathematically and computationally correct, the problem lives in the real world, and in the real world there are a number of complexities that we can exploit to shift the equation to our advantage. the first important point is that human memory is not a computer-like store of data; the extent to which you can accurately remember information often depends on how you memorize it, and in what format you store it. for example, we implicitly memorize kilobytes of information fairly easily in the form of human faces, but even something as similar in the grand scheme of things as dog faces are much harder for us. information in the form of text is even harder although if we memorize the text visually and orally at the same time it's somewhat easier again. some have tried to take advantage of this fact by generating random brainwallets and encoding them in a sequence of words; for example, one might see something like: witch collapse practice feed shame open despair creek road again ice least a popular xkcd comic illustrates the principle, suggesting that users create passwords by generating four random words instead of trying to be clever with symbol manipulation. the approach seems elegant, and perhaps taking away of our differing ability to remember random symbols and language in this way, it just might work. except, there's a problem: it doesn't. to quote a recent study by richard shay and others from carnegie mellon: in a 1,476-participant online study, we explored the usability of 3and 4-word systemassigned passphrases in comparison to system-assigned passwords composed of 5 to 6 random characters, and 8-character system-assigned pronounceable passwords. contrary to expectations, system-assigned passphrases performed similarly to system-assigned passwords of similar entropy across the usability metrics we examined. passphrases and passwords were forgotten at similar rates, led to similar levels of user difficulty and annoyance, and were both written down by a majority of participants. however, passphrases took significantly longer for participants to enter, and appear to require error-correction to counteract entry mistakes. passphrase usability did not seem to increase when we shrunk the dictionary from which words were chosen, reduced the number of words in a passphrase, or allowed users to change the order of words. however, the paper does leave off on a note of hope. it does note that there are ways to make passwords that are higher entropy, and thus higher security, while still being just as easy to memorize; randomly generated but pronounceable strings like "zelactudet" (presumably created via some kind of per-character language model sampling) seem to provide a moderate gain over both word lists and randomly generated character strings. a likely cause of this is that pronounceable passwords are likely to be memorized both as a sound and as a sequence of letters, increasing redundancy. thus, we have at least one strategy for improving memorizability without sacrificing strength. the other strategy is to attack the problem from the opposite end: make it harder to crack the password without increasing entropy. we cannot make the password harder to crack by adding more combinations, as that would increase entropy, but what we can do is use what is known as a hard key derivation function. for example, suppose that if our memorized brainwallet is b, instead of making the private key sha256(b) or sha3(b), we make it f(b, 1000) where f is defined as follows: def f(b, rounds): x = b i = 0 while i < rounds: x = sha3(x + b) i += 1 return x essentially, we keep feeding b into the hash function over and over again, and only after 1000 rounds do we take the output. feeding the original input back into each round is not strictly necessary, but cryptographers recommend it in order to limit the effect of attacks involving precomputed rainbow tables. now, checking each individual password takes a thousand time longer. you, as the legitimate user, won't notice the difference it's 20 milliseconds instead of 20 microseconds but against attackers you get ten bits of entropy for free, without having to memorize anything more. if you go up to 30000 rounds you get fifteen bits of entropy, but then calculating the password takes close to a second; 20 bits takes 20 seconds, and beyond about 23 it becomes too long to be practical. now, there is one clever way we can go even further: outsourceable ultra-expensive kdfs. the idea is to come up with a function which is extremely expensive to compute (eg. 240 computational steps), but which can be computed in some way without giving the entity computing the function access to the output. the cleanest, but most cryptographically complicated, way of doing this is to have a function which can somehow be "blinded" so unblind(f(blind(x))) = f(x) and blinding and unblinding requires a one-time randomly generated secret. you then calculate blind(password), and ship the work off to a third party, ideally with an asic, and then unblind the response when you receive it. one example of this is using elliptic curve cryptography: generate a weak curve where the values are only 80 bits long instead of 256, and make the hard problem a discrete logarithm computation. that is, we calculate a value x by taking the hash of a value, find the associated y on the curve, then we "blind" the (x,y) point by adding another randomly generated point, n (whose associated private key we know to be n), and then ship the result off to a server to crack. once the server comes up with the private key corresponding to n + (x,y), we subtract n, and we get the private key corresponding to (x,y) our intended result. the server does not learn any information about what this value, or even (x,y), is theoretically it could be anything with the right blinding factor n. also, note that the user can instantly verify the work simply convert the private key you get back into a point, and make sure that the point is actually (x,y). another approach relies somewhat less on algebraic features of nonstandard and deliberately weak elliptic curves: use hashes to derive 20 seeds from a password, apply a very hard proof of work problem to each one (eg. calculate f(h) = n where n is such that sha3(n+h) < 2^216), and combine the values using a moderately hard kdf at the end. unless all 20 servers collude (which can be avoided if the user connects through tor, since it would be impossible even for an attacker controlling or seeing the results of 100% of the network to determine which requests are coming from the same user), the protocol is secure. the interesting thing about both of these protocols is that they are fairly easy to turn into a "useful proof of work" consensus algorithm for a blockchain; anyone could submit work for the chain to process, the chain would perform the computations, and both elliptic curve discrete logs and hash-based proofs of work are very easy to verify. the elegant part of the scheme is that it turns to social use both users' expenses in computing the work function, but also attackers' much greater expenses. if the blockchain subsidized the proof of work, then it would be optimal for attackers to also try to crack users' passwords by submitting work to the blockchain, in which case the attackers would contribute to the consensus security in the process. but then, in reality at this level of security, where 240 work is needed to compute a single password, brainwallets and other passwords would be so secure that no one would even bother attacking them. entropy differentials now, we get to our final, and most interesting, memorization strategy. from what we discussed above, we know that entropy, the amount of information in a message, and the complexity of attack are exactly identical unless you make the process deliberately slower with expensive kdfs. however, there is another point about entropy that was mentioned in passing, and which is actually crucial: experienced entropy is context-dependent. the name "mahmoud ahmadjinejad" might have perhaps ten to fifteen bits of entropy to us, but to someone living in iran while he was president it might have only four bits in the list of the most important people in their lives, he is quite likely in the top sixteen. your parents or spouse are completely unknown to myself, and so for me their names have perhaps twenty bits of entropy, but to you they have only two or three bits. why does this happen? formally, the best way to think about it is that for each person the prior experiences of their lives create a kind of compression algorithm, and under different compression algorithms, or different programming languages, the same string can have a different kolmogorov complexity. in python, '111111111111111111' is just '1'*18, but in javascript it's array(19).join("1"). in a hypothetical version of python with the variable x preset to '111111111111111111', it's just x. the last example, although seemingly contrived, is actually the one that best describes much of the real world; the human mind is a machine with many variables preset by our past experiences. this rather simple insight leads to a particularly elegant strategy for password memorizability: try to create a password where the "entropy differential", the difference between the entropy to you and the entropy to other people, is as large as possible. one simple strategy is to prepend your own username to the password. if my password were to be "yui&(4_", i might do "vbuterin:yui&(4_" instead. my username might have about ten to fifteen bits of entropy to the rest of the world, but to me it's almost a single bit. this is essentially the primary reason why usernames exist as an account protection mechanism alongside passwords even in cases where the concept of users having "names" is not strictly necessary. now, we can go a bit further. one common piece of advice that is now commonly and universally derided as worthless is to pick a password by taking a phrase out of a book or song. the reason why this idea is seductive is because it seems to cleverly exploit differentials: the phrase might have over 100 bits of entropy, but you only need to remember the book and the page and line number. the problem is, of course, that everyone else has access to the books as well, and they can simply do a brute force attack over all books, songs and movies using that information. however, the advice is not worthless; in fact, if used as only part of your password, a quote from a book, song or movie is an excellent ingredient. why? simple: it creates a differential. your favorite line from your favorite song only has a few bits of entropy to you, but it's not everyone's favorite song, so to the entire world it might have ten or twenty bits of entropy. the optimal strategy is thus to pick a book or song that you really like, but which is also maximally obscure push your entropy down, and others' entropy higher. and then, of course, prepend your username and append some random characters (perhaps even a random pronounceable "word" like "zelactudet"), and use a secure kdf. conclusion how much entropy do you need to be secure? right now, password cracking chips can perform about 236 attempts per second, and bitcoin miners can perform roughly 240 hashes per second (that's 1 terahash). the entire bitcoin network together does 250 petahashes, or about 257 hashes per second. cryptographers generally consider 280 to be an acceptable minimum level of security. to get 80 bits of entropy, you need either about 17 random letters of the alphabet, or 12 random letters, numbers and symbols. however, we can shave quite a bit off the requirement: fifteen bits for a username, fifteen bits for a good kdf, perhaps ten bits for an abbreviation from a passage from a semi-obscure song or book that you like, and then 40 more bits of plan old simple randomness. if you're not using a good kdf, then feel free to use other ingredients. it has become rather popular among security experts to dismiss passwords as being fundamentally insecure, and argue for password schemes to be replaced outright. a common argument is that because of moore's law attackers' power increases by one bit of entropy every two years, so you will have to keep on memorizing more and more to remain secure. however, this is not quite correct. if you use a hard kdf, moore's law allows you to take away bits from the attacker's power just as quickly as the attacker gains power, and the fact that schemes such as those described above, with the exception of kdfs (the moderate kind, not the outsourceable kind), have not even been tried suggests that there is still some way to go. on the whole, passwords thus remain as secure as they have ever been, and remain very useful as one ingredient of a strong security policy just not the only ingredient. moderate approaches that use a combination of hardware wallets, trusted third parties and brainwallets may even be what wins out in the end. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements gav’s ethereum ðξv update iv | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search gav’s ethereum ðξv update iv posted by gavin wood on december 15, 2014 research & development time for another update! so quite a bit has happened following ðξvcon-0, our internal developer's conference. the conference itself was a great time to get all the developers together and really get to know each other, dissipate a lot of information (back to back presentations for 5 days!) and chat over a lot of ideas. the comms team will be releasing each of the presentations as fast as ian can get them nicely polished. during the time since the last update, much has happened including, finally, the release of the ethereum ðξv website, ethdev.com. though relatively simple as present, there are great plans to extend this into a developer's portal in which you'll be able to browse the bug bounty programme, look at and, ultimately follow tutorials, look up documentation, find the latest binaries for each platform and see the progress of builds. as usual i have been mostly between switzerland, the uk and berlin, during this time. now that ðξv-berlin is settled in the hub, we have a great collaboration space in which volunteers can work, collaborate, bond and socialise alongside our more formal hires. of late, i have been working to finish up the formal specification of ethereum, the yellow paper, and make it up to date with the latest protocol changes in order that the security audit get underway. together we have been putting the finishing touches on seventh, and likely final, proof-of-concept code, delayed largely due to a desire to make it the final poc release for protocol changes. i've also been doing some nice core refactoring and documentation, specifically removing two long standing dislikes of mine, the state::create and state::call methods and making the state class nicer for creating custom states useful when developing contracts. you can expect to see the fruits of this work in milestone ii of mix, ethereum's official ide. ongoing recruitment on that note, i'm happy to announce that we have hired arkadiy paronyan, a talented developer originally from russia who will be working with yann on the mix ide. he's got off to a great start on his first week helping on the front-end with the second milestone. i'm also very pleased to announce that we hired gustav simonsson. being an expert erlang with go experience with considerable expertise in network programming and security reviewing, he will initially be working with jutta on the go code base security audit before joining the go team. we also have another two recruits: dimitri khoklov and jason colby. i first met jason in the fateful week back last january when the early ethereum collaborators got together for a week before the north american bitcoin conference where vitalik gave the first public talk about ethereum. jason, who has moved to berlin from his home in new hampshire, is mostly working alongside aeron and christian to help to look after the hub and looking after various bits of administration that need to be done. dimitri, who works from tver in russia is helping flesh out our unit tests with christoph, ultimately aiming towards full code coverage. we have several more recruits that i'd love to mention but can't announce quite yet watch this space... (: ongoing projects i'm happy to say that after a busy weekend, marek, caktux, nick and sven have managed to get the build bot, our ci system, building on all three platforms cleanly again. a special shout goes out to marek who tirelessly fought with cmake and msvc to bend the windows platform to his will. well done to all involved. christian continues to power through on the solidity project, aided now by lefteris who is focusing on parsing and packaging the natspec documentation. the latest feature to be added allows for the creation of new contracts in a beautiful manner with the new keyword. alex and sven are beginning to work on the project of introducing network well-formedness into the p2p subsystem using the salient parts of the well-proven kademlia dht design. we should begin seeing some of this stuff in the code base within before the year-end. i'm also happy to announce that the first successful message was sent between go & c++ clients on our messaging/hash-table hybrid system, codenamed whisper. though only at an early proof-of-concept stage, the api is reasonably robust and fixed, so largely ready to prototype applications on. new projects marian is the lucky guy who has been tasked with developing out what will be our awesome web-based c&c deck. this will provide a public website whose back-end connects to a bunch of nodes around the world and displays real-time information on network status including chain length and a chain-fork early warning system. though accessible by anyone, we will of course have a dedicated monitor on at all times for this page at the hub. sven, jutta and heiko have also begun a most interesting and important project: the ethereum stress-testing project. designed to study and test the network in a range of real-life adverse situations prior to release, they will construct infrastructure allowing the setup of many (10s, 100s, even 1000s of) nodes each individually remote-controllable and able to simulate circumstances such as isp attacks, net splits, rogue clients, arrival and departure of large amounts of hash-power and measure attributes like block & transaction propagation times and patterns, uncle rates and fork lengths. a project to watch out for. conclusions the next time i write this i hope to have released poc-7 and be on the way to the alpha release (not to mention have the yellow paper out). i expect jeff will be doing an update concerning the go side of things soon enough. until then, watch out for the poc-7 release and mine some testnet ether! previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements scalability, part 1: building on top | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search scalability, part 1: building on top posted by vitalik buterin on september 17, 2014 research & development over the next few weeks, i am going to make a series of posts that is going to be a large overview of the possibilities for scalability of ethereum, intending to create a precise understanding of the problems at bay in implementing a scalable cryptocurrency infrastructure, and where the least-bad tradeoffs and sacrifices required to solve those problems might lie. as a general outline of the form that this series is going to take, i intend to first discuss the fundamental problem with ethereum 1.0 as it stands, as well as every other cryptocurrency platform in existence, and introduce limited solutions to specific problems that allow for much more efficiency in some cases increasing efficiency by a constant factor, and in other cases making a more fundamental complexity-theoretic improvement but only in very specific use cases. in later posts, i will discuss further and further generalizations of such mechanisms, and finally culminating in the ultimate generalization: applying the tactics that i describe to make certain programs run better inside of ethereum to ethereum itself providing at least one route to ethereum 2.0. fundamentally, the problem of scaling up something like bitcoin and ethereum is an extremely hard one; the consensus architectures strongly rely on every node processing every transaction, and they do so in a very deep way. there do exist protocols for "light clients" to work with ethereum, storing only a small part of the blockchain and using merkle trees to securely access the rest, but even still the network relies on a relatively large number of full nodes to achieve high degrees of security. scaling up to visa or swift levels of transaction volume is possible, but only at the cost of sacrificing decentralization as only a very small number of full nodes will survive. if we want to reach such levels, and go even higher with micropayments, we need to develop a consensus architecture which achieves a fundamental improvement over "every node processing every transaction". however, as it turns out, there is a lot that we can do without going that far. protocol enhancements image from https://bitcoin.org/en/developer-guide the first step in increasing space efficiency is some structural alterations to the protocol alterations that have already been part of ethereum since day one. the first is a shift from utxo-based architecture to account-based architecture. the bitcoin blockchain relies on a concept of "unspent transaction outputs" every transaction contains one or more inputs and one or more outputs, with the condition that each input must reference a valid and unspent previous output and the total sum of the outputs must be no greater than the total sum of the inputs. this requires transactions to be large, often containing multiple signatures from the same user, and requires about 50 bytes to be stored in the database for every transaction that a node receives. it is particularly inconvenient when you have an account that very many people are sending small payments to; in the case of ethereum.org, it will take us hundreds of transactions to clear our exodus address. ripple and ethereum instead use a more conventional system of transactions depositing to and withdrawing from accounts, ensuring that each account takes up only about 100 bytes on the blockchain regardless of its level of usage. a second protocol adjustment, used by both ripple and ethereum, is that of storing the full blockchain state in a patricia tree in every block. the patricia tree structure is designed to include maximal deduplication, so if you are storing many nearly-identical patricia trees for consecutive blocks you only need to store most of the data once. this allows nodes to more easily "start from the middle" and securely download the current state without having to process the entire history. these schemes are, of course, counterbalanced by the fact that ethereum opens itself up to a wider array of applications and thus a much more active array of usage, and at the end of the day such optimizations can only go so far. thus, to go further, we need to go beyond tweaks to the protocol itself, and build on top. batching in bitcoin, one transaction that spends ten previously unspent outputs requires ten signatures. in ethereum, one transaction always requires one signature (although in the case of constructions like multisig accounts multiple transactions may be needed to process a withdrawal). however, one can go even further, and create a system where ten withdrawals only require one transaction and one signature. this is another constant-factor improvement, but a potentially rather powerful one: batching. the idea behind batching is simple: put multiple sends into a single transaction in the data fields, and then have a forwarding contract split up the payment. here is the simple implementation of such a contract: i = 0 while i < msg.datasize: send(msg.data[i], msg.data[i+1]) i += 2 we can also extend it to support forwarding messages, using some low-level evm commands in serpent to do some byte-by-byte packing: init: contract.storage[0] = msg.sender code: if msg.sender != contract.storage[0]: stop i = 0 while i < ~calldatasize(): to = ~calldataload(i) value = ~calldataload(i+20) / 256^12 datasize = ~calldataload(i+32) / 256^30 data = alloc(datasize) ~calldatacopy(data, i+34, datasize) ~call(tx.gas 25, to, value, data, datasize, 0, 0) i += 34 + datasize instead of using your normal account to interact with contracts, the idea is that you would store your funds and maintain your relationships with contracts using this account, and then you will be able to make as many operations as you need all at once with a single transaction. note that this scheme does have its limits. although it can arbitrarily magnify the amount of work that can be done with one signature, the amount of data that must be spent registering the recipient, value and message data, and the amount of computational resources that must be spent processing the transactions, still remains the same. the importance of signatures is not to be underestimated; signature verification is likely the most expensive part of blockchain validation, but the efficiency gain from using this kind of mechanism is still limited to perhaps something like a factor of four for plain old sends, and even less for transactions that involve a lot of computation. micropayment channels a common dream application of cryptocurrency is the idea of micropayments having markets on very tiny chunks of computational or physical resources, paying for electricity, internet bandwidth, file storage, road usage or any other micro-meterable good one cent at a time. existing cryptocurrencies are certainly useful for much smaller payments than were possible before; paypal charges a fixed fee of $0.30 per transaction, and bitcoin currently charges ~$0.05, making it logical to send payments as low as 50 cents in size. however, if we want to pay $0.01 at a time, then we need a much better scheme. there is no easy universal scheme to implement; if there was, that would be ethereum 2.0. rather, there is a combination of different approaches, where each approach is suited for a particular use case. one common use case is micropayment channels: situations where one party is paying the other over time for a metered service (eg. a file download), and the transaction only needs to be processed at the end. bitcoin supports micropayment channels; ethererum does as well, and arguably somewhat more elegantly. the channel works roughly as follows: the sender sends a transaction to initialize a channel, specifying a recipient, and the contract initializes a channel with value zero and supplies an id for the channel. to increase the payment on the channel, the sender signs a data packet of the form [id, value], with value being the new value to transmit. when the channel process is done, and the recipient wants to cash out, he must simply take the signed [id, value, v, r, s] packet (the v,r,s triple being an elliptic curve signature) and push it to the blockchain as transaction data, and the contract verifies the signature. if the signature is valid, the contract waits 1000 blocks for a higher-valued packet for the transaction id to be sent, and can then be pinged again to send the funds. note that if the sender tries to cheat by submitting an earlier packet with a low value, the receiver has the 1000 block interval to submit the higher-valued packet. the code for the validator is as follows: # create channel: [0, to] if msg.data[0] == 0: new_id = contract.storage[-1] # store [from, to, value, maxvalue, timeout] in contract storage contract.storage[new_id] = msg.sender contract.storage[new_id + 1] = msg.data[1] contract.storage[new_id + 2] = 0 contract.storage[new_id + 3] = msg.value contract.storage[new_id + 4] = 2^254 # increment next id contract.storage[-1] = new_id + 10 # return id of this channel return(new_id) elif msg.data[0] == 2: id = msg.data[1] % 2^160 # check if timeout has run out if block.number >= contract.storage[id + 3]: # send funds send(contract.storage[id + 1], contract.storage[id + 2]) # send refund send(contract.storage[id], contract.storage[id + 3] contract.storage[id + 2]) # clear storage contract.storage[id] = 0 contract.storage[id + 1] = 0 contract.storage[id + 2] = 0 contract.storage[id + 3] = 0 contract.storage[id + 4] = 0 and there we go. all that is needed now is a decent off-chain user interface for processing the consumer-merchant side of the transaction. probabilistic micropayments but even still, micropayment channels are not a panacea. what if you only need to pay $0.007 to download a 32 mb file from someone, so even the entire transaction is not worth the single final transaction fee? for this, we do something slightly more clever: probabilistic micropayments. essentially, a probabilistic micropayment occurs when a sender performs an action which provably has a specified probability of allowing a certain payment to happen in the future; here, we might do a 0.7% chance of paying $1. in the long term, both expenses and receipts will be roughly the same as in the non-probabilistic model, but with the benefit of saving 99% on transaction fees. so, how do we do probabilistic micropayments? the general approach is to have the payment be a signed data packet of the form [nonce, timeout, to, value, prob], where nonce is a random number, timeout is a near-future block number, to is the recipient, value is the amount of ether to send and prob is the probability of sending multiplied by 232, and then when the block number surpasses timeout allow the data packet to be supplied to the blockchain and cashed out only if a random number generator, seeded with the nonce, supplies a value which mod 232 is less than prob. assuming a random number generator, the code snippet for the basic receiving function is: # cash out: [0, nonce, timeout, to, value, prob, v, r, s] if msg.data[0] == 0: # helper contracts (addresses obviously won't work on testnet or livenet) ecrecover = 0x46a8d0b21b1336d83b06829f568d7450df36883f random = 0xb7d0a063fafca596de6af7b5062926c0f793c7db # variables timeout = msg.data[2] to = msg.data[3] value = msg.data[4] prob = msg.data[5] # is it time to cash out? if block.number >= timeout: # randomness if call(random, [0, nonce, timeout], 3) % 2^32 < msg.data[5]: # determine sender h = sha3(slice(msg.data, 1), 5) sender = call(ecrecover, [h, msg.data[6], msg.data[7], msg.data[8]], 4) # withdraw if contract.storage[sender] >= value: contract.storage[sender] -= value send(to, value) there are two "hard parts" in the implementation of this approach. one is double-spending attacks, and the other is how to build the random number generator. to defeat double-spending attacks, the strategy is simple: require a very high security deposit in the contract alongside the account's ether balance available for sending. if the sendable balance drops below zero, destroy the entire deposit. the second part is, of course, how to build a random number generator in the first place. generally, the main source of randomness used in ethereum is block hashes; because micropayments are low-value applications, and because the different nonce on each transaction ensures that a block hash is extremely unlikely to favor any particular user in any particular way, block hashes will likely be sufficient for this purpose however, we need to make sure we grab a specific block hash rather than simply the block hash when a request is sent (using the block hash when a request is sent also works, but less well, since the sender and receiver have an incentive to try to disrupt each other's attempts to send claim transactions during blocks that are unfavorable to them). one option is to have a centralized contract maintain a list of the block hash for every block, incentivizing miners to ping it every block; the contract can charge a micropayment for its api in order to pay for the service. for efficiency, one can limit the contract to providing a reward once every ten blocks. in the event that the contract skips over a block, the next block hash is used. the code for the one-every-ten-blocks version is: # if we get pinged for the first time in a new epoch, set the prevhash if !contract.storage[block.number / 10]: send(msg.sender, 10^17) contract.storage[block.number / 10] = block.prevhash # otherwise, provide the block hash: [0, block number] if msg.data == 0 and msg.value > 10^16: return(contract.storage[msg.data[1] / 10]) in order to convert this into a suitable implementation of the random contract, we just do: # if we get pinged for the first time in a new epoch, set the prevhash if !contract.storage[block.number / 10]: send(msg.sender, 10^17) contract.storage[block.number / 10] = block.prevhash # otherwise, provide the hash of the block hash plus a nonce: [0, block number, nonce] if msg.data == 0 and msg.value > 10^16: return(sha3([contract.storage[msg.data[1] / 10], msg.data[2]], 2)) note that for something like this to work efficiently, one "higher-level" piece of infrastructure that needs to exist is some kind of incentivized pinging. this job can be done cooperatively with a pub/sub contract: a contract can be made which other contracts subscribe to, paying a very small fee, and when the contract gets pinged for the first time in n blocks it provides a single reward and immediately pings all of the contracts that subscribed to it. this strategy is still vulnerable to some abuse by miners, but the low-value nature of micropayments and the independence of each payment should limit the problem drastically. off-chain oracles following the spirit of signature batching, an approach that goes even further is to take the entire computation off the blockchain. in order to do so securely, we use a clever economic hack: the code still goes on the blockchain, and gets recorded there, but by default the computation is decided by oracles which run the code off-chain in a private evm and supply the answer, also providing a security deposit. when the answer is supplied, it takes 100 blocks until the answer is committed; if everything goes well, the answer can be committed to the blockchain after 100 blocks, and the oracle recovers its deposit and a small bonus. however, within that 100-block interval, any node can check the computation themselves, and if they see that the oracle is wrong they can pay for an auditing transaction essentially, actually run the code on the blockchain, and see if the result turns out to be the same. if it does not, then the auditor gets 90% of the block reward and the other 10% is destroyed. essentially, this provides near-equivalent assurances to every node running the code, except that in practice only a few nodes do. particularly, if there is a financial contract, the parties to the financial contract have a strong incentive to carry out the audit, because they are the ones who would be screwed over by an invalid block. this scheme is elegant, but somewhat inconvenient; it requires users to wait 100 blocks before the results of their code can be used. to solve that problem, the protocol can be extended even further. now, the idea is to create an entire "shadow chain", with computations happening off-chain but state transitions being committed back to the main chain after 100 blocks. oracles can add new blocks to the "tail" of the chain, where a block consists of a list of transactions and a [[k1, v1], [k2, v2] ... ] list of state transitions caused by those transactions. if a block is unchallenged for 100 blocks, the state transitions are applied automatically to the main chain. if the block is successfully challenged before it is committed then that block and all children are reverted, and the block and all children lose their deposits with part going to the auditor and part to the void (note that this creates extra incentive to audit, since now the author of the child of a shadow block would prefer to audit that shadow block lest they be caught up in the author's potential malfeasance). the code for this is much more complicated than the other examples; a complete but untested version can be found here. note that this protocol is still a limited one: it solves the signature verification problem, and it solves the state transition computation problem, but it still does not solve the data problem. every transaction in this model must still be downloaded by every node. how do we do even better? as it turns out, we probably can; however, to go further than this we have to solve a much larger problem: the problem of data. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements why not just use x? an instructive example from bitcoin | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search why not just use x? an instructive example from bitcoin posted by vitalik buterin on february 9, 2014 research & development bitcoin developer gregory maxwell writes the following on reddit: there is a design flaw in the bitcoin protocol where its possible for a third party to take a valid transaction of yours and mutate it in a way which leaves it valid and functionally identical but with a different transaction id. this greatly complicates writing correct wallet software, and it can be used abusively to invalidate long chains of unconfirmed transactions that depend on the non-mutant transaction (since transactions refer to each other by txid). this issue arises from several sources, one of them being openssl’s willingness to accept and make sense of signatures with invalid encodings. a normal ecdsa signature encodes two large integers, the encoding isn’t constant length— if there are leading zeros you are supposed to drop them. it’s easy to write software that assumes the signature will be a constant length and then leave extra leading zeros in them. this is a very interesting cautionary tale, and is particularly important because situations like these are part of the reason why we have made certain design decisions in our development philosophy. specifically, the issue is this: many people continue to bring up the point that we are in many places unnecessarily reinventing the wheel, creating our own serialization format, rlp, instead of using the existing protobuf and we’re building an application-specific scripting language instead of “just using lua”. this is a very valid concern; not-invented-here syndrome is a commonly-used pejorative, so doing such in-house development does require justification. and the cautionary tale i quoted above provides precisely the perfect example of the justification that i will provide. external technologies, whether protobuf, lua or openssl, are very good, and have years of development behind them, but in many cases they were never designed with the perfect consensus, determinism and cryptographic integrity in mind that cryptocurrencies require. the openssl situation above is the perfect example; aside from cryptocurrencies, there really is no other situations where the fact that you can take a valid signature and turn it into another valid signature with a different hash is a significant problem, and yet here it’s fatal. one of our core principles in ethereum is simplicity; the protocol should be as simple as possible, and the protocol should not contain any black boxes. every single feature of every single sub-protocol should be precisely 100% documented on the whitepaper or wiki, and implemented using that as a specification (ie. test-driven development). doing this for an existing software package is arguably almost as hard as building an entirely new package from scratch; in fact, it may even be harder, since existing software packages often have more complexity than they need to in order to be feature-complete, whereas our alternatives do not – read the protobuf spec and compare it to the rlp spec to understand what i mean. note that the above principle has its limits. for example, we are certainly not foolish enough to start inventing our own hash algorithms, instead using the universally acclaimed and well-vetted sha3, and for signatures we’re using the same old secp256k1 as bitcoin, although we’re using rlp to store the v,r,s triple (the v is an extra two bits for public key recovery purposes) instead of the openssl buffer protocol. these kinds of situations are the ones where “just using x” is precisely the right thing to do, because x has a clean and well-understood interface and there are no subtle differences between different implementations. the sha3 of the empty string is c5d2460186...a470 in c++, in python, and in javascript; there’s no debate about it. in between these two extremes, it’s basically a matter of finding the right balance. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements proof of stake: how i learned to love weak subjectivity | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search proof of stake: how i learned to love weak subjectivity posted by vitalik buterin on november 25, 2014 research & development proof of stake continues to be one of the most controversial discussions in the cryptocurrency space. although the idea has many undeniable benefits, including efficiency, a larger security margin and future-proof immunity to hardware centralization concerns, proof of stake algorithms tend to be substantially more complex than proof of work-based alternatives, and there is a large amount of skepticism that proof of stake can work at all, particularly with regard to the supposedly fundamental "nothing at stake" problem. as it turns out, however, the problems are solvable, and one can make a rigorous argument that proof of stake, with all its benefits, can be made to be successful but at a moderate cost. the purpose of this post will be to explain exactly what this cost is, and how its impact can be minimized. economic sets and nothing at stake first, an introduction. the purpose of a consensus algorithm, in general, is to allow for the secure updating of a state according to some specific state transition rules, where the right to perform the state transitions is distributed among some economic set. an economic set is a set of users which can be given the right to collectively perform transitions via some algorithm, and the important property that the economic set used for consensus needs to have is that it must be securely decentralized meaning that no single actor, or colluding set of actors, can take up the majority of the set, even if the actor has a fairly large amount of capital and financial incentive. so far, we know of three securely decentralized economic sets, and each economic set corresponds to a set of consensus algorithms: owners of computing power: standard proof of work, or tapow. note that this comes in specialized hardware, and (hopefully) general-purpose hardware variants. stakeholders: all of the many variants of proof of stake a user's social network: ripple/stellar-style consensus note that there have been some recent attempts to develop consensus algorithms based on traditional byzantine fault tolerance theory; however, all such approaches are based on an m-of-n security model, and the concept of "byzantine fault tolerance" by itself still leaves open the question of which set the n should be sampled from. in most cases, the set used is stakeholders, so we will treat such neo-bft paradigms are simply being clever subcategories of "proof of stake". proof of work has a nice property that makes it much simpler to design effective algorithms for it: participation in the economic set requires the consumption of a resource external to the system. this means that, when contributing one's work to the blockchain, a miner must make the choice of which of all possible forks to contribute to (or whether to try to start a new fork), and the different options are mutually exclusive. double-voting, including double-voting where the second vote is made many years after the first, is unprofitablem since it requires you to split your mining power among the different votes; the dominant strategy is always to put your mining power exclusively on the fork that you think is most likely to win. with proof of stake, however, the situation is different. although inclusion into the economic set may be costly (although as we will see it not always is), voting is free. this means that "naive proof of stake" algorithms, which simply try to copy proof of work by making every coin a "simulated mining rig" with a certain chance per second of making the account that owns it usable for signing a block, have a fatal flaw: if there are multiple forks, the optimal strategy is to vote on all forks at once. this is the core of "nothing at stake". note that there is one argument for why it might not make sense for a user to vote on one fork in a proof-of-stake environment: "altruism-prime". altruism-prime is essentially the combination of actual altruism (on the part of users or software developers), expressed both as a direct concern for the welfare of others and the network and a psychological moral disincentive against doing something that is obviously evil (double-voting), as well as the "fake altruism" that occurs because holders of coins have a desire not to see the value of their coins go down. unfortunately, altruism-prime cannot be relied on exclusively, because the value of coins arising from protocol integrity is a public good and will thus be undersupplied (eg. if there are 1000 stakeholders, and each of their activity has a 1% chance of being "pivotal" in contributing to a successful attack that will knock coin value down to zero, then each stakeholder will accept a bribe equal to only 1% of their holdings). in the case of a distribution equivalent to the ethereum genesis block, depending on how you estimate the probability of each user being pivotal, the required quantity of bribes would be equal to somewhere between 0.3% and 8.6% of total stake (or even less if an attack is nonfatal to the currency). however, altruism-prime is still an important concept that algorithm designers should keep in mind, so as to take maximal advantage of in case it works well. short and long range if we focus our attention specifically on short-range forks forks lasting less than some number of blocks, perhaps 3000, then there actually is a solution to the nothing at stake problem: security deposits. in order to be eligible to receive a reward for voting on a block, the user must put down a security deposit, and if the user is caught either voting on multiple forks then a proof of that transaction can be put into the original chain, taking the reward away. hence, voting for only a single fork once again becomes the dominant strategy. another set of strategies, called "slasher 2.0" (in contrast to slasher 1.0, the original security deposit-based proof of stake algorithm), involves simply penalizing voters that vote on the wrong fork, not voters that double-vote. this makes analysis substantially simpler, as it removes the need to pre-select voters many blocks in advance to prevent probabilistic double-voting strategies, although it does have the cost that users may be unwilling to sign anything if there are two alternatives of a block at a given height. if we want to give users the option to sign in such circumstances, a variant of logarithmic scoring rules can be used (see here for more detailed investigation). for the purposes of this discussion, slasher 1.0 and slasher 2.0 have identical properties. the reason why this only works for short-range forks is simple: the user has to have the right to withdraw the security deposit eventually, and once the deposit is withdrawn there is no longer any incentive not to vote on a long-range fork starting far back in time using those coins. one class of strategies that attempt to deal with this is making the deposit permanent, but these approaches have a problem of their own: unless the value of a coin constantly grows so as to continually admit new signers, the consensus set ends up ossifying into a sort of permanent nobility. given that one of the main ideological grievances that has led to cryptocurrency's popularity is precisely the fact that centralization tends to ossify into nobilities that retain permanent power, copying such a property will likely be unacceptable to most users, at least for blockchains that are meant to be permanent. a nobility model may well be precisely the correct approach for special-purpose ephemeral blockchains that are meant to die quickly (eg. one might imagine such a blockchain existing for a round of a blockchain-based game). one class of approaches at solving the problem is to combine the slasher mechanism described above for short-range forks with a backup, transactions-as-proof-of-stake, for long range forks. tapos essentially works by counting transaction fees as part of a block's "score" (and requiring every transaction to include some bytes of a recent block hash to make transactions not trivially transferable), the theory being that a successful attack fork must spend a large quantity of fees catching up. however, this hybrid approach has a fundamental flaw: if we assume that the probability of an attack succeeding is near-zero, then every signer has an incentive to offer a service of re-signing all of their transactions onto a new blockchain in exchange for a small fee; hence, a zero probability of attacks succeeding is not game-theoretically stable. does every user setting up their own node.js webapp to accept bribes sound unrealistic? well, if so, there's a much easier way of doing it: sell old, no-longer-used, private keys on the black market. even without black markets, a proof of stake system would forever be under the threat of the individuals that originally participated in the pre-sale and had a share of genesis block issuance eventually finding each other and coming together to launch a fork. because of all the arguments above, we can safely conclude that this threat of an attacker building up a fork from arbitrarily long range is unfortunately fundamental, and in all non-degenerate implementations the issue is fatal to a proof of stake algorithm's success in the proof of work security model. however, we can get around this fundamental barrier with a slight, but nevertheless fundamental, change in the security model. weak subjectivity although there are many ways to categorize consensus algorithms, the division that we will focus on for the rest of this discussion is the following. first, we will provide the two most common paradigms today: objective: a new node coming onto the network with no knowledge except (i) the protocol definition and (ii) the set of all blocks and other "important" messages that have been published can independently come to the exact same conclusion as the rest of the network on the current state. subjective: the system has stable states where different nodes come to different conclusions, and a large amount of social information (ie. reputation) is required in order to participate. systems that use social networks as their consensus set (eg. ripple) are all necessarily subjective; a new node that knows nothing but the protocol and the data can be convinced by an attacker that their 100000 nodes are trustworthy, and without reputation there is no way to deal with that attack. proof of work, on the other hand, is objective: the current state is always the state that contains the highest expected amount of proof of work. now, for proof of stake, we will add a third paradigm: weakly subjective: a new node coming onto the network with no knowledge except (i) the protocol definition, (ii) the set of all blocks and other "important" messages that have been published and (iii) a state from less than n blocks ago that is known to be valid can independently come to the exact same conclusion as the rest of the network on the current state, unless there is an attacker that permanently has more than x percent control over the consensus set. under this model, we can clearly see how proof of stake works perfectly fine: we simply forbid nodes from reverting more than n blocks, and set n to be the security deposit length. that is to say, if state s has been valid and has become an ancestor of at least n valid states, then from that point on no state s' which is not a descendant of s can be valid. long-range attacks are no longer a problem, for the trivial reason that we have simply said that long-range forks are invalid as part of the protocol definition. this rule clearly is weakly subjective, with the added bonus that x = 100% (ie. no attack can cause permanent disruption unless it lasts more than n blocks). another weakly subjective scoring method is exponential subjective scoring, defined as follows: every state s maintains a "score" and a "gravity" score(genesis) = 0, gravity(genesis) = 1 score(block) = score(block.parent) + weight(block) * gravity(block.parent), where weight(block) is usually 1, though more advanced weight functions can also be used (eg. in bitcoin, weight(block) = block.difficulty can work well) if a node sees a new block b' with b as parent, then if n is the length of the longest chain of descendants from b at that time, gravity(b') = gravity(b) * 0.99 ^ n (note that values other than 0.99 can also be used). essentially, we explicitly penalize forks that come later. ess has the property that, unlike more naive approaches at subjectivity, it mostly avoids permanent network splits; if the time between the first node on the network hearing about block b and the last node on the network hearing about block b is an interval of k blocks, then a fork is unsustainable unless the lengths of the two forks remain forever within roughly k percent of each other (if that is the case, then the differing gravities of the forks will ensure that half of the network will forever see one fork as higher-scoring and the other half will support the other fork). hence, ess is weakly subjective with x roughly corresponding to how close to a 50/50 network split the attacker can create (eg. if the attacker can create a 70/30 split, then x = 0.29). in general, the "max revert n blocks" rule is superior and less complex, but ess may prove to make more sense in situations where users are fine with high degrees of subjectivity (ie. n being small) in exchange for a rapid ascent to very high degrees of security (ie. immune to a 99% attack after n blocks). consequences so what would a world powered by weakly subjective consensus look like? first of all, nodes that are always online would be fine; in those cases weak subjectivity is by definition equivalent to objectivity. nodes that pop online once in a while, or at least once every n blocks, would also be fine, because they would be able to constantly get an updated state of the network. however, new nodes joining the network, and nodes that appear online after a very long time, would not have the consensus algorithm reliably protecting them. fortunately, for them, the solution is simple: the first time they sign up, and every time they stay offline for a very very long time, they need only get a recent block hash from a friend, a blockchain explorer, or simply their software provider, and paste it into their blockchain client as a "checkpoint". they will then be able to securely update their view of the current state from there. this security assumption, the idea of "getting a block hash from a friend", may seem unrigorous to many; bitcoin developers often make the point that if the solution to long-range attacks is some alternative deciding mechanism x, then the security of the blockchain ultimately depends on x, and so the algorithm is in reality no more secure than using x directly implying that most x, including our social-consensus-driven approach, are insecure. however, this logic ignores why consensus algorithms exist in the first place. consensus is a social process, and human beings are fairly good at engaging in consensus on our own without any help from algorithms; perhaps the best example is the rai stones, where a tribe in yap essentially maintained a blockchain recording changes to the ownership of stones (used as a bitcoin-like zero-intrinsic-value asset) as part of its collective memory. the reason why consensus algorithms are needed is, quite simply, because humans do not have infinite computational power, and prefer to rely on software agents to maintain consensus for us. software agents are very smart, in the sense that they can maintain consensus on extremely large states with extremely complex rulesets with perfect precision, but they are also very ignorant, in the sense that they have very little social information, and the challenge of consensus algorithms is that of creating an algorithm that requires as little input of social information as possible. weak subjectivity is exactly the correct solution. it solves the long-range problems with proof of stake by relying on human-driven social information, but leaves to a consensus algorithm the role of increasing the speed of consensus from many weeks to twelve seconds and of allowing the use of highly complex rulesets and a large state. the role of human-driven consensus is relegated to maintaining consensus on block hashes over long periods of time, something which people are perfectly good at. a hypothetical oppressive government which is powerful enough to actually cause confusion over the true value of a block hash from one year ago would also be powerful enough to overpower any proof of work algorithm, or cause confusion about the rules of blockchain protocol. note that we do not need to fix n; theoretically, we can come up with an algorithm that allows users to keep their deposits locked down for longer than n blocks, and users can then take advantage of those deposits to get a much more fine-grained reading of their security level. for example, if a user has not logged in since t blocks ago, and 23% of deposits have term length greater than t, then the user can come up with their own subjective scoring function that ignores signatures with newer deposits, and thereby be secure against attacks with up to 11.5% of total stake. an increasing interest rate curve can be used to incentivize longer-term deposits over shorter ones, or for simplicity we can just rely on altruism-prime. marginal cost: the other objection one objection to long-term deposits is that it incentivizes users to keep their capital locked up, which is inefficient, the exact same problem as proof of work. however, there are four counterpoints to this. first, marginal cost is not total cost, and the ratio of total cost divided by marginal cost is much less for proof of stake than proof of work. a user will likely experience close to no pain from locking up 50% of their capital for a few months, a slight amount of pain from locking up 70%, but would find locking up more than 85% intolerable without a large reward. additionally, different users have very different preferences for how willing they are to lock up capital. because of these two factors put together, regardless of what the equilibrium interest rate ends up being, the vast majority of the capital will be locked up at far below marginal cost. second, locking up capital is a private cost, but also a public good. the presence of locked up capital means that there is less money supply available for transactional purposes, and so the value of the currency will increase, redistributing the capital to everyone else, creating a social benefit. third, security deposits are a very safe store of value, so (i) they substitute the use of money as a personal crisis insurance tool, and (ii) many users will be able to take out loans in the same currency collateralized by the security deposit. finally, because proof of stake can actually take away deposits for misbehaving, and not just rewards, it is capable of achieving a level of security much higher than the level of rewards, whereas in the case of proof of work the level of security can only equal the level of rewards. there is no way for a proof of work protocol to destroy misbehaving miners' asics. fortunately, there is a way to test those assumptions: launch a proof of stake coin with a stake reward of 1%, 2%, 3%, etc per year, and see just how large a percentage of coins become deposits in each case. users will not act against their own interests, so we can simply use the quantity of funds spent on consensus as a proxy for how much inefficiency the consensus algorithm introduces; if proof of stake has a reasonable level of security at a much lower reward level than proof of work, then we know that proof of stake is a more efficient consensus mechanism, and we can use the levels of participation at different reward levels to get an accurate idea of the ratio between total cost and marginal cost. ultimately, it may take years to get an exact idea of just how large the capital lockup costs are. altogether, we now know for certain that (i) proof of stake algorithms can be made secure, and weak subjectivity is both sufficient and necessary as a fundamental change in the security model to sidestep nothing-at-stake concerns to accomplish this goal, and (ii) there are substantial economic reasons to believe that proof of stake actually is much more economically efficient than proof of work. proof of stake is not an unknown; the past six months of formalization and research have determined exactly where the strengths and weaknesses lie, at least to as large extent as with proof of work, where mining centralization uncertainties may well forever abound. now, it's simply a matter of standardizing the algorithms, and giving blockchain developers the choice. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle in-person meatspace protocol to prove unconditional possession of a private key 2019 oct 01 see all posts recommended pre-reading: https://ethresear.ch/t/minimal-anti-collusion-infrastructure/5413 alice slowly walks down the old, dusty stairs of the building into the basement. she thinks wistfully of the old days, when quadratic-voting in the world collective market was a much simpler process of linking her public key to a twitter account and opening up metamask to start firing off votes. of course back then voting in the wcm was used for little; there were a few internet forums that used it for voting on posts, and a few million dollars donated to its quadratic funding oracle. but then it grew, and then the game-theoretic attacks came. first came the exchange platforms, which started offering "dividends" to anyone who registered a public key belonging to an exchange and thus provably allowed the exchange to vote on their behalf, breaking the crucial "independent choice" assumption of the quadratic voting and funding mechanisms. and soon after that came the fake accounts twitter accounts, reddit accounts filtered by karma score, national government ids, all proved vulnerable to either government cheating or hackers, or both. elaborate infrastructure was instituted at registration time to ensure both that account holders were real people, and that account holders themselves held the keys, not a central custody service purchasing keys by the thousands to buy votes. and so today, voting is still easy, but initiation, while still not harder than going to a government office, is no longer exactly trivial. but of course, with billions of dollars in donations from now-deceased billionaires and cryptocurrency premines forming part of the wcm's quadratic funding pool, and elements of municipal governance using its quadratic voting protocols, participating is very much worth it. after reaching the end of the stairs, alice opens the door and enters the room. inside the room, she sees a table. on the near side of the table, she sees a single, empty chair. on the far side of the table, she sees four people already sitting down on chairs of their own, the high-reputation guardians randomly selected by the wcm for alice's registration ceremony. "hello, alice," the person sitting on the leftmost chair, whose name she intuits is bob, says in a calm voice. "glad that you can make it," the person sitting beside bob, whose name she intuits is charlie, adds. alice walks over to the chair that is clearly meant for her and sits down. "let us begin," the person sitting beside charlie, whose name by logical progression is david, proclaims. "alice, do you have your key shares?" alice takes out four pocket-sized notebooks, clearly bought from a dollar store, and places them on the table. the person sitting at the right, logically named evan, takes out his phone, and immediately the others take out theirs. they open up their ethereum wallets. "so," evan begins, "the current ethereum beacon chain slot number is 28,205,913, and the block hash starts 0xbe48. do all agree?". "yes," alice, bob, charlie and david exclaim in unison. evan continues: "so let us wait for the next block." the five intently stare at their phones. first for ten seconds, then twenty, then thirty. "three skipped proposers," bob mutters, "how unusual". but then after another ten seconds, a new block appears. "slot number 28,205,917, block hash starts 0x62f9, so first digit 6. all agreed?" "yes." "six mod four is two, and as is prescribed in the old ways, we start counting indices from zero, so this means alice will keep the third book, counting as usual from our left." bob takes the first, second and fourth notebooks that alice provided, leaving the third untouched. alice takes the remaining notebook and puts it back in her backpack. bob opens each notebook to a page in the middle with the corner folded, and sees a sequence of letters and numbers written with a pencil in the middle of each page a standard way of writing the key shares for over a decade, since camera and image processing technology got powerful enough to recognize words and numbers written on single slips of paper even inside an envelope. bob, charlie, david and evan crowd around the books together, and each open up an app on their phone and press a few buttons. bob starts reading, as all four start typing into their phones at the same time: "alice's first key share is, 6-b-d-7-h-k-k-l-o-e-q-q-p-3-y-s-6-x-e-f. applying the 100,000x iterated sha256 hash we get e-a-6-6..., confirm?" "confirmed," the others replied. "checking against alice's precommitted elliptic curve point a0... match." "alice's second key share is, f-r-n-m-j-t-x-r-s-3-b-u-n-n-n-i-z-3-d-g. iterated hash 8-0-3-c..., confirm?" "confirmed. checking against alice's precommitted elliptic curve point a1... match." "alice's fourth key share is, i-o-f-s-a-q-f-n-w-f-6-c-e-a-m-s-6-z-z-n. iterated hash 6-a-5-6..., confirm?" "confirmed. checking against alice's precommitted elliptic curve point a3... match." "adding the four precommitted curve points, x coordinate begins 3-1-8-3. alice, confirm that that is the key you wish to register?" "confirm." bob, charlie, david and evan glance down at their smartphone apps one more time, and each tap a few buttons. alice catches a glance at charlie's phone; she sees four yellow checkmarks, and an "approval transaction pending" dialog. after a few seconds, the four yellow checkmarks are replaced with a single green checkmark, with a transaction hash id, too small for alice to make out the digits from a few meters away, below. alice's phone soon buzzes, with a notification dialog saying "registration confirmed". "congratulations, alice," bob says. "unconditional possession of your key has been verified. you are now free to send a transaction to the world collective market's mpc oracle to update your key." "only a 75% probability this would have actually caught me if i didn't actually have all four parts of the key," alice thought to herself. but it seemed to be enough for an in-person protocol in practice; and if it ever wasn't then they could easily switch to slightly more complex protocols that used low-degree polynomials to achieve exponentially high levels of soundness. alice taps a few buttons on her smartphone, and a "transaction pending" dialog shows up on the screen. five seconds later, the dialog disappears and is replaced by a green checkmark. she jumps up with joy and, before bob, charlie, david and evan can say goodbye, runs out of the room, frantically tapping buttons to vote on all the projects and issues in the wcm that she had wanted to support for months. the search for a stable cryptocurrency | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the search for a stable cryptocurrency posted by vitalik buterin on november 11, 2014 research & development special thanks to robert sams for the development of seignorage shares and insights regarding how to correctly value volatile coins in multi-currency systems note: we are not planning on adding price stabilization to ether; our philosophy has always been to keep ether simple to minimize black-swan risks. results of this research will likely go into either subcurrencies or independent blockchains one of the main problems with bitcoin for ordinary users is that, while the network may be a great way of sending payments, with lower transaction costs, much more expansive global reach, and a very high level of censorship resistance, bitcoin the currency is a very volatile means of storing value. although the currency had by and large grown by leaps and bounds over the past six years, especially in financial markets past performance is no guarantee (and by efficient market hypothesis not even an indicator) of future results of expected value, and the currency also has an established reputation for extreme volatility; over the past eleven months, bitcoin holders have lost about 67% of their wealth and quite often the price moves up or down by as much as 25% in a single week. seeing this concern, there is a growing interest in a simple question: can we get the best of both worlds? can we have the full decentralization that a cryptographic payment network offers, but at the same time have a higher level of price stability, without such extreme upward and downward swings? last week, a team of japanese researchers made a proposal for an "improved bitcoin", which was an attempt to do just that: whereas bitcoin has a fixed supply, and a volatile price, the researchers' improved bitcoin would vary its supply in an attempt to mitigate the shocks in price. however, the problem of making a price-stable cryptocurrency, as the researchers realized, is much different from that of simply setting up an inflation target for a central bank. the underlying question is more difficult: how do we target a fixed price in a way that is both decentralized and robust against attack? to resolve the issue properly, it is best to break it down into two mostly separate sub-problems: how do we measure a currency's price in a decentralized way? given a desired supply adjustment to target the price, to whom do we issue and how do we absorb currency units? decentralized measurement for the decentralized measurement problem, there are two known major classes of solutions: exogenous solutions, mechanisms which try to measure the price with respect to some precise index from the outside, and endogenous solutions, mechanisms which try to use internal variables of the network to measure price. as far as exogenous solutions go, so far the only reliable known class of mechanisms for (possibly) cryptoeconomically securely determining the value of an exogenous variable are the different variants of schellingcoin essentially, have everyone vote on what the result is (using some set chosen randomly based on mining power or stake in some currency to prevent sybil attacks), and reward everyone that provides a result that is close to the majority consensus. if you assume that everyone else will provide accurate information, then it is in your interest to provide accurate information in order to be closer to the consensus a self-reinforcing mechanism much like cryptocurrency consensus itself. the main problem with schellingcoin is that it's not clear exactly how stable the consensus is. particularly, what if some medium-sized actor pre-announces some alternative value to the truth that would be beneficial for most actors to adopt, and the actors manage to coordinate on switching over? if there was a large incentive, and if the pool of users was relatively centralized, it might not be too difficult to coordinate on switching over. there are three major factors that can influence the extent of this vulnerability: is it likely that the participants in a schellingcoin actually have a common incentive to bias the result in some direction? do the participants have some common stake in the system that would be devalued if the system were to be dishonest? is it possible to "credibly commit" to a particular answer (ie. commit to providing the answer in a way that obviously can't be changed)? (1) is rather problematic for single-currency systems, as if the set of participants is chosen by their stake in the currency then they have a strong incentive to pretend the currency price is lower so that the compensation mechanism will push it up, and if the set of participants is chosen by mining power then they have a strong incentive to pretend the currency's price is too high so as to increase the issuance. now, if there are two kinds of mining, one of which is used to select schellingcoin participants and the other to receive a variable reward, then this objection no longer applies, and multi-currency systems can also get around the problem. (2) is true if the participant selection is based on either stake (ideally, long-term bonded stake) or asic mining, but false for cpu mining. however, we should not simply count on this incentive to outweigh (1). (3) is perhaps the hardest; it depends on the precise technical implementation of the schellingcoin. a simple implementation involving simply submitting the values to the blockchain is problematic because simply submitting one's value early is a credible commitment. the original schellingcoin used a mechanism of having everyone submit a hash of the value in the first round, and the actual value in the second round, sort of a cryptographic equivalent to requiring everyone to put down a card face down first, and then flip it at the same time; however, this too allows credible commitment by revealing (even if not submitting) one's value early, as the value can be checked against the hash. a third option is requiring all of the participants to submit their values directly, but only during a specific block; if a participant does release a submission early they can always "double-spend" it. the 12-second block time would mean that there is almost no time for coordination. the creator of the block can be strongly incentivized (or even, if the schellingcoin is an independent blockchain, required) to include all participations, to discourage or prevent the block maker from picking and choosing answers. a fourth class of options involves some secret sharing or secure multiparty computation mechanism, using a collection of nodes, themselves selected by stake (perhaps even the participants themselves), as a sort of decentralized substitute for a centralized server solution, with all the privacy that such an approach entails. finally, a fifth strategy is to do the schellingcoin "blockchain-style": every period, some random stakeholder is selected, and told to provide their vote as a [id, value] pair, where value is the actual valid and id is an identifier of the previous vote that looks correct. the incentive to vote correctly is that only tests that remain in the main chain after some number of blocks are rewarded, and future voters will note attach their vote to a vote that is incorrect fearing that if they do voters after them will reject their vote. schellingcoin is an untested experiment, and so there is legitimate reason to be skeptical that it will work; however, if we want anything close to a perfect price measurement scheme it's currently the only mechanism that we have. if schellingcoin proves unworkable, then we will have to make do with the other kinds of strategies: the endogenous ones. endogenous solutions to measure the price of a currency endogenously, what we essentially need is to find some service inside the network that is known to have a roughly stable real-value price, and measure the price of that service inside the network as measured in the network's own token. examples of such services include: computation (measured via mining difficulty) transaction fees data storage bandwidth provision a slightly different, but related, strategy, is to measure some statistic that correllates indirectly with price, usually a metric of the level of usage; one example of this is transaction volume. the problem with all of these services is, however, that none of them are very robust against rapid changes due to technological innovation. moore's law has so far guaranteed that most forms of computational services become cheaper at a rate of 2x every two years, and it could easily speed up to 2x every 18 months or 2x every five years. hence, trying to peg a currency to any of those variables will likely lead to a system which is hyperinflationary, and so we need some more advanced strategies for using these variables to determine a more stable metric of the price. first, let us set up the problem. formally, we define an estimator to be a function which receives a data feed of some input variable (eg. mining difficulty, transaction cost in currency units, etc) d[1], d[2], d[3]..., and needs to output a stream of estimates of the currency's price, p[1], p[2], p[3]... the estimator obviously cannot look into the future; p[i] can be dependent on d[1], d[2] ... d[i], but not d[i+1]. now, to start off, let us graph the simplest possible estimator on bitcoin, which we'll call the naive estimator: difficulty equals price. unfortunately, the problem with this approach is obvious from the graph and was already mentioned above: difficulty is a function of both price and moore's law, and so it gives results that depart from any accurate measure of the price exponentially over time. the first immediate strategy to fix this problem is to try to compensate for moore's law, using the difficulty but artificially reducing the price by some constant per day to counteract the expected speed of technological progress; we'll call this the compensated naive estimator. note that there are an infinite number of versions of this estimator, one for each depreciation rate, and all of the other estimators that we show here will also have parameters. the way that we will select the parameter for our version is by using a variant of simulated annealing to find the optimal values, using the first 780 days of the bitcoin price as "training data". the estimators are then left to perform as they would for the remaining 780 days, to see how they would react to conditions that were unknown when the parameters were optimized (this technique, knows as "cross-validation", is standard in machine learning and optimization theory). the optimal value for the compensated estimator is a drop of 0.48% per day, leading to this chart: the next estimator that we will explore is the bounded estimator. the way the bounded estimator works is somewhat more complicated. by default, it assumes that all growth in difficulty is due to moore's law. however, it assumes that moore's law cannot go backwards (ie. technology getting worse), and that moore's law cannot go faster than some rate in the case of our version, 5.88% per two weeks, or roughly quadrupling every year. any growth outside these bounds it assumes is coming from price rises or drops. thus, for example, if the difficulty rises by 20% in one period, it assumes that 5.88% of it is due to technological advancements, and the remaining 14.12% is due to a price increase, and thus a stabilizing currency based on this estimator might increase supply by 14.12% to compensate. the theory is that cryptocurrency price growth to a large extent happens in rapid bubbles, and thus the bounded estimator should be able to capture the bulk of the price growth during such events. there are more advanced strategies as well; the best strategies should take into account the fact that asic farms take time to set up, and also follow a hysteresis effect: it's often viable to keep an asic farm online if you already have it even when under the same conditions it would not be viable to start up a new one. a simple approach is looking at the rate of increase of the difficulty, and not just the difficulty itself, or even using a linear regression analysis to project difficulty 90 days into the future. here is a chart containing the above estimators, plus a few others, compared to the actual price: note that the chart also includes three estimators that use statistics other than bitcoin mining: a simple and an advanced estimator using transaction volume, and an estimator using the average transaction fee. we can also split up the mining-based estimators from the other estimators: see https://github.com/ethereum/economic-modeling/tree/master/stability for the source code that produced these results. of course, this is only the beginning of endogenous price estimator theory; a more thorough analysis involving dozens of cryptocurrencies will likely go much further. the best estimators may well end up using a combination of different measures; seeing how the difficulty-based estimators overshot the price in 2014 and the transaction-based estimators undershot the price, the two combined could end up being substantially more accurate. the problem is also going to get easier over time as we see the bitcoin mining economy stabilize toward something closer to an equilibrium where technology improves only as fast as the general moore's law rule of 2x every 2 years. to see just how good these estimators can get, we can note from the charts that they can cancel out at least 50% of cryptocurrency price volatility, and may increase to ~67% once the mining industry stabilizes. something like bitcoin, if it becomes mainstream, will likely be somewhat more unstable than gold, but not that much more unstable the only difference between btc and gold is that the supply of gold can actually increase as the price goes higher since more can be mined if miners are willing to pay higher costs, so there is an implicit dampening effect, but the supply elasticity of gold is surprisingly not that high; production barely increased at all during the run-ups in price during the 1970s and 2000s. the price of gold stayed within a range of 4.63x ($412 to $1980) in the last decade; logarithmically reducing that by two thirds gives a range of 1.54x, not much higher than eur/usd (1.37x), jpy/usd (1.64x) or cad/usd (1.41x); thus, endogenous stabilization may well prove quite viable, and may be preferred due to its lack of tie to any specific centralized currency or authority. the other issue that all of these estimators have to contend with is exploitability: if transaction volume is used to determine the currency's price, then an attacker can manipulate the price very easily by simply sending very many transactions. the average transaction fees paid in bitcoin are about $5000 per day; at that price in a stabilized currency the attacker would be able to halve the price. mining difficulty, however, is much more difficult to exploit simply because the market is so large. if a platform does not want to accept the inefficiencies of wasteful proof of work, an alternative is to build in a market for other resources, such as storage, instead; filecoin and permacoin are two efforts that attempt to use a decentralized file storage market as a consensus mechanism, and the same market could easily be dual-purposed to serve as an estimator. the issuance problem now, even if we have a reasonably good, or even perfect, estimator for the currency's price, we still have the second problem: how do we issue or absorb currency units? the simplest approach is to simply issue them as a mining reward, as proposed by the japanese researchers. however, this has two problems: such a mechanism can only issue new currency units when the price is too high; it cannot absorb currency units when the price is too low. if we are using mining difficulty in an endogenous estimator, then the estimator needs to take into account the fact that some of the increases in mining difficulty will be a result of an increased issuance rate triggered by the estimator itself. if not handled very carefully, the second problem has the potential to create some rather dangerous feedback loops in either direction; however, if we use a different market as an estimator and as an issuance model then this will not be a problem. the first problem seems serious; in fact, one can interpret it as saying that any currency using this model will always be strictly worse than bitcoin, because bitcoin will eventually have an issuance rate of zero and a currency using this mechanism will have an issuance rate always above zero. hence, the currency will always be more inflationary, and thus less attractive to hold. however, this argument is not quite true; the reason is that when a user purchases units of the stabilized currency then they have more confidence that at the time of purchase the units are not already overvalued and therefore will soon decline. alternatively, one can note that extremely large swings in price are justified by changing estimations of the probability the currency will become thousands of times more expensive; clipping off this possibility will reduce the upward and downward extent of these swings. for users who care about stability, this risk reduction may well outweigh the increased general long-term supply inflation. bitassets a second approach is the (original implementation of the) "bitassets" strategy used by bitshares. this approach can be described as follows: there exist two currencies, "vol-coins" and "stable-coins". stable-coins are understood to have a value of $1. vol-coins are an actual currency; users can have a zero or positive balance of them. stable-coins exist only in the form of contracts-for-difference (ie. every negative stable-coin is really a debt to someone else, collateralized by at least 2x the value in vol-coins, and every positive stable-coin is the ownership of that debt). if the value of someone's stable-coin debt exceeds 90% of the value of their vol-coin collateral, the debt is cancelled and the entire vol-coin collateral is transferred to the counterparty ("margin call") users are free to trade vol-coins and stable-coins with each other. and that's it. the key piece that makes the mechanism (supposedly) work is the concept of a "market peg": because everyone understands that stable-coins are supposed to be worth $1, if the value of a stable-coin drops below $1, then everyone will realize that it will eventually go back to $1, and so people will buy it, so it actually will go back to $1 a self-fulfilling prophecy argument. and for a similar reason, if the price goes above $1, it will go back down. because stable-coins are a zero-total-supply currency (ie. each positive unit is matched by a corresponding negative unit), the mechanism is not intrinsically unworkable; a price of $1 could be stable with ten users or ten billion users (remember, fridges are users too!). however, the mechanism has some rather serious fragility properties. sure, if the price of a stable-coin goes to $0.95, and it's a small drop that can easily be corrected, then the mechanism will come into play, and the price will quickly go back to $1. however, if the price suddenly drops to $0.90, or lower, then users may interpret the drop as a sign that the peg is actually breaking, and will start scrambling to get out while they can thus making the price fall even further. at the end, the stable-coin could easily end up being worth nothing at all. in the real world, markets do often show positive feedback loops, and it is quite likely that the only reason the system has not fallen apart already is because everyone knows that there exists a large centralized organization (bitshares inc) which is willing to act as a buyer of last resort to maintain the "market" peg if necessary. note that bitshares has now moved to a somewhat different model involving price feeds provided by the delegates (participants in the consensus algorithm) of the system; hence the fragility risks are likely substantially lower now. schellingdollar an approach vaguely similar to bitassets that arguably works much better is the schellingdollar (called that way because it was originally intended to work with the schellingcoin price detection mechanism, but it can also be used with endogenous estimators), defined as follows: there exist two currencies, "vol-coins" and "stable-coins". vol-coins are initially distributed somehow (eg. pre-sale), but initially no stable-coins exist. users may have only a zero or positive balance of vol-coins. users may have a negative balance of stable-coins, but can only acquire or increase their negative balance of stable-coins if they have a quantity of vol-coins equal in value to twice their new stable-coin balance (eg. if a stable-coin is $1 and a vol-coin is $5, then if a user has 10 vol-coins ($50) they can at most reduce their stable-coin balance to -25) if the value of a user's negative stable-coins exceeds 90% of the value of the user's vol-coins, then the user's stable-coin and vol-coin balances are both reduced to zero ("margin call"). this prevents situations where accounts exist with negative-valued balances and the system goes bankrupt as users run away from their debt. users can convert their stable-coins into vol-coins or their vol-coins into stable-coins at a rate of $1 worth of vol-coin per stable-coin, perhaps with a 0.1% exchange fee. this mechanism is of course subject to the limits described in (2). the system keeps track of the total quantity of stable-coins in circulation. if the quantity exceeds zero, the system imposes a negative interest rate to make positive stable-coin holdings less attractive and negative holdings more attractive. if the quantity is less than zero, the system similarly imposes a positive interest rate. interest rates can be adjusted via something like a pid controller, or even a simple "increase or decrease by 0.2% every day based on whether the quantity is positive or negative" rule. here, we do not simply assume that the market will keep the price at $1; instead, we use a central-bank-style interest rate targeting mechanism to artificially discourage holding stable-coin units if the supply is too high (ie. greater than zero), and encourage holding stable-coin units if the supply is too low (ie. less than zero). note that there are still fragility risks here. first, if the vol-coin price falls by more than 50% very quickly, then many margin call conditions will be triggered, drastically shifting the stable-coin supply to the positive side, and thus forcing a high negative interest rate on stable-coins. second, if the vol-coin market is too thin, then it will be easily manipulable, allowing attackers to trigger margin call cascades. another concern is, why would vol-coins be valuable? scarcity alone will not provide much value, since vol-coins are inferior to stable-coins for transactional purposes. we can see the answer by modeling the system as a sort of decentralized corporation, where "making profits" is equivalent to absorbing vol-coins and "taking losses" is equivalent to issuing vol-coins. the system's profit and loss scenarios are as follows: profit: transaction fees from exchanging stable-coins for vol-coins profit: the extra 10% in margin call situations loss: situations where the vol-coin price falls while the total stable-coin supply is positive, or rises while the total stable-coin supply is negative (the first case is more likely to happen, due to margin-call situations) profit: situations where the vol-coin price rises while the total stable-coin supply is positive, or falls while it's negative note that the second profit is in some ways a phantom profit; when users hold vol-coins, they will need to take into account the risk that they will be on the receiving end of this extra 10% seizure, which cancels out the benefit to the system from the profit existing. however, one might argue that because of the dunning-kruger effect users might underestimate their susceptibility to eating the loss, and thus the compensation will be less than 100%. now, consider a strategy where a user tries to hold on to a constant percentage of all vol-coins. when x% of vol-coins are absorbed, the user sells off x% of their vol-coins and takes a profit, and when new vol-coins equal to x% of the existing supply are released, the user increases their holdings by the same portion, taking a loss. thus, the user's net profit is proportional to the total profit of the system. seignorage shares a fourth model is "seignorage shares", courtesy of robert sams. seignorage shares is a rather elegant scheme that, in my own simplified take on the scheme, works as follows: there exist two currencies, "vol-coins" and "stable-coins" (sams uses "shares" and "coins", respectively) anyone can purchase vol-coins for stable-coins or vol-coins for stable-coins from the system at a rate of $1 worth of vol-coin per stable-coin, perhaps with a 0.1% exchange fee note that in sams' version, an auction was used to sell off newly-created stable-coins if the price goes too high, and buy if it goes too low; this mechanism basically has the same effect, except using an always-available fixed price in place of an auction. however, the simplicity comes at the cost of some degree of fragility. to see why, let us make a similar valuation analysis for vol-coins. the profit and loss scenarios are simple: profit: absorbing vol-coins to issue new stable-coins loss: issuing vol-coins to absorb stable-coins the same valuation strategy applies as in the other case, so we can see that the value of the vol-coins is proportional to the expected total future increase in the supply of stable-coins, adjusted by some discounting factor. thus, here lies the problem: if the system is understood by all parties to be "winding down" (eg. users are abandoning it for a superior competitor), and thus the total stable-coin supply is expected to go down and never come back up, then the value of the vol-coins drops below zero, so vol-coins hyperinflate, and then stable-coins hyperinflate. in exchange for this fragility risk, however, vol-coins can achieve a much higher valuation, so the scheme is much more attractive to cryptoplatform developers looking to earn revenue via a token sale. note that both the schellingdollar and seignorage shares, if they are on an independent network, also need to take into account transaction fees and consensus costs. fortunately, with proof of stake, it should be possible to make consensus cheaper than transaction fees, in which case the difference can be added to profits. this potentially allows for a larger market cap for the schellingdollar's vol-coin, and allows the market cap of seignorage shares' vol-coins to remain above zero even in the event of a substantial, albeit not total, permanent decrease in stable-coin volume. ultimately, however, some degree of fragility is inevitable: at the very least, if interest in a system drops to near-zero, then the system can be double-spent and estimators and schellingcoins exploited to death. even sidechains, as a scheme for preserving one currency across multiple networks, are susceptible to this problem. the question is simply (1) how do we minimize the risks, and (2) given that risks exist, how do we present the system to users so that they do not become overly dependent on something that could break? conclusions are stable-value assets necessary? given the high level of interest in "blockchain technology" coupled with disinterest in "bitcoin the currency" that we see among so many in the mainstream world, perhaps the time is ripe for stable-currency or multi-currency systems to take over. there would then be multiple separate classes of cryptoassets: stable assets for trading, speculative assets for investment, and bitcoin itself may well serve as a unique schelling point for a universal fallback asset, similar to the current and historical functioning of gold. if that were to happen, and particularly if the stronger version of price stability based on schellingcoin strategies could take off, the cryptocurrency landscape may end up in an interesting situation: there may be thousands of cryptocurrencies, of which many would be volatile, but many others would be stable-coins, all adjusting prices nearly in lockstep with each other; hence, the situation could even end up being expressed in interfaces as a single super-currency, but where different blockchains randomly give positive or negative interest rates, much like ferdinando ametrano's "hayek money". the true cryptoeconomy of the future may have not even begun to take shape. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements daos are not scary, part 2: reducing barriers | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search daos are not scary, part 2: reducing barriers posted by vitalik buterin on march 1, 2014 research & development in the last installment of this series, we talked about what “smart contracts” (or, perhaps more accurately, “self-enforcing contracts”) are, and discussed in detail the two main mechanisms through which these contracts can have “force”: smart property and “factum” currencies. we also discussed the limits of smart contracts, and how a smart contract-enabled legal system might use a combination of human judgement and automatic execution to achieve the best possible outcomes. but what is the point of these contracts? why automate? why is it better to have our relationships regulated and controlled by algorithms rather than humans? these are the tough questions that this article, and the next, intends to tackle. a tale of two industries the first, and most obvious, benefit of using internet-driven technology to automate anything is the exact same that we have seen the internet, and bitcoin, already provide in the spheres of communications and commerce: it increases efficiency and reduces barriers to entry. one very good example of this effect providing meaningful benefits in the traditional world is the publishing industry. in the 1970s, if you wanted to write a book, there was a large number of opaque, centralized intermediaries that you would need to go through before your book would get to a consumer. first, you would need a publishing company, which would also handle editing and marketing for you and provide a quality control function to the consumer. second, the book would need to be distributed, and then finally it would be sold at each individual bookstore. each part of the chain would take a large cut; at the end, you would be lucky to get more than ten percent of the revenue from each copy as a royalty. notice the use of the term “royalty”, implying that you the author of the book are simply just another extraneous part of the chain that deserves a few percent as a cut rather than, well, the single most important person without whom the book would not even exist in the first place. now, the situation is greatly improved. we now have distinct printing companies, marketing companies and bookstores, with a clear and defined role for each one and plenty of competition in each industry – and if you’re okay with keeping it purely digital, you can just publish on kindle and get 70%. now, let’s consider a very similar example, but with a completely different industry: consumer protection, or more specifically escrow. escrow is a very important function in commerce, and especially commerce online; when you buy a product from a small online store or from a merchant on ebay, you are participating in a transaction where neither side has a substantial reputation, and so when you send the money by default there is no way to be sure that you will actually get anything to show for it. escrow provides the solution: instead of sending the money to the merchant directly, you first send the money to an escrow agent, and the escrow agent then waits for you to confirm that you received the item. if you confirm, then the escrow agent sends the money along, and if the merchant confirms that they can’t send the item then the escrow agent gives you your money back. if there’s a dispute, an adjudication process begins, and the escrow agent decides which side has the better case. the way it’s implemented today, however, escrow is handled by centralized entities, and is thrown in together with a large number of other functions. on the online marketplace ebay, for example, ebay serves the role of providing a server for the seller to host their product page on, a search and price comparison function for products, and a rating system for buyers and sellers. ebay also owns paypal, which actually moves the money from the seller to the buyer and serves as the escrow agent. essentially, this is exactly the same situation that book publishing was in in the 1970s, although in fairness to ebay sellers do get quite a bit more than 10% of their money. so how can we make an ideal marketplace with cryptocurrencies and smart contracts? if we wanted to be extreme about it, we could make the marketplace decentralized, using a diaspora-like model to allow a seller to host their products on a specialized site, on their own server or on a decentralized dropbox implementation, use a namecoin-like system for sellers to store their identities and keep a web of trust on the blockchain. however, what we’re looking at now is a more moderate and simple goal: separating out the function of the escrow agent from the payment system. fortunately, bitcoin offers a solution: multisignature transactions. introducing multisig multisignature transactions allow a user to send funds to an address with three private keys, such that you need two of those keys to unlock the funds (multisigs can also be 1-of-3, 6-of-9, or anything else, but in practice 2-of-3 is the most useful). the way to apply this to escrow is simple: create a 2-of-3 escrow between the buyer, the seller and the escrow agent, have the buyer send funds into it and when a transaction is complete the buyer and the seller sign a transaction to complete the escrow. if there is a dispute, the escrow agent picks which side has the more convincing case, and signs a transaction with them to send them the funds. on a technological level, this is slightly complicated, but fortunately bitrated has come up with a site that makes the process quite easy for the average user. of course, in its current form, bitrated is not perfect, and we do not see that much bitcoin commerce using it. the interface is arguably not as easy as it could be, especially since most people are not used to the idea of storing specific per-transaction links for a few weeks, and it would be much more powerful if it was integrated into a fully-fledged merchant package. one design might be a kryptokit-like web app, showing each user a list of “open” buys and sells and providing a “finalize”, “accept”, “cancel” and “dispute” button for each one; users would then be able to interact with the multisig system just as if it was a standard payment processor, but then get a notification to finalize or dispute their purchases after a few weeks. but if bitrated does get its interface right and starts to see mass adoption, what will that accomplish? once again, the answer is reduced barriers to entry. currently, getting into the consumer escrow and arbitration business is hard. in order to be an escrow service, you essentially need to build an entire platform and an ecosystem, so that consumers and merchants operate through you. you also can’t just be the one escrowing the money – you also need to be the one transferring the money in the first place. ebay needs to have, and control, paypal, in order for half of its consumer protection to work. with bitrated, this all changes. anyone can become an escrow agent and arbitrator, and an ebay-like marketplace (perhaps cryptothrift or the upcoming egora) can have a rating system for arbitrators as well as buyers and sellers. alternatively, the system could handle arbitration in the background similarly to how uber handles taxi drivers: anyone could become an arbitrator after a vetting process, and the system would automatically reward arbitrators with good ratings and fire those with bad ratings. fees would drop, likely substantially below even the 2.9% charged by paypal alone. smart contracts smart contracts in general take this same basic idea, and push it much further. instead of relying on a platform like bitfinex to hedge one’s bitcoin holdings or speculate in either direction at high leverage, one can use a blockchain-based financial derivatives contract with a decentralized order book, leaving no central party to take any fees. the ongoing cost of maintaining an exchange, complete with operational security, server management, ddos protection, marketing and legal expenses, could be replaced with a one-time effort to write the contract, likely in less than 100 lines of code, and another one-time effort to make a pretty interface. from that point on, the entire system would be free except for network fees. file storage platforms like dropbox could be similarly replaced; although, since hard disk space costs money, the system would not be free, it would likely be substantially cheaper than it is today. it would also help equalize the market by making it easy to participate on the supply side: anyone with a big hard drive, or even a small hard drive with some extra space, can simply install the app and start earning money renting out their unused space. instead of relying on legal contracts using expensive (and often, especially in international circumstances and poor countries, ineffective) court systems, or even moderately expensive private arbitration services, business relationships can be governed by smart contracts where those parts of the contract that do need human interpretation can be segregated into many specialized parts. there might be judges specializing in determining whether or not a product shipped (ideally, this would be the postal system itself), judges specializing in determining whether web application designs meet specifications, judges specializing in adjudicating certain classes of property insurance claims with a $0.75 fee by examining satellite images, and there would be contract writers skilled in intelligently integrating each one. specialization has its advantages, and is the reason why society moved beyond running after bears with stone clubs and picking berries, but one of its weaknesses has always been the fact that it requires intermediaries to manage and function, including intermediaries specifically to manage the relationship between the intermediaries. smart contracts can remove the latter category almost completely, allowing for an even greater degree of specialization, along with lower barriers to entry within each now shrunken category. however, this increase in efficiency is only one part of the puzzle. the other part, and perhaps the more important one, has to do with a topic that many cryptocurrency advocates hold dear: reducing trust. we will cover that in the next installment of this series. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ether purchase troubleshooting | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ether purchase troubleshooting posted by vitalik buterin on july 23, 2014 organizational although we hope that the ether purchasing experience goes smoothly for everyone, we recognize that there will always be situations where things do not quite go as planned. perhaps your internet connection dies in the middle of your purchase. perhaps you accidentally click a back button or some link or simply hit refresh while the purchase is in progress. perhaps you forgot to download your wallet. perhaps you think you may have forgotten your password, and you want to make sure you have it down correctly. for all of these situations, the user experience is unfortunately going to be a bit more tricky than simply downloading a web app; a bit of command line action with a python script will be required. first of all, let's go over downloading the python script. to get the script installed, download the zip archive from here, and unpack it. then, navigate to the directory, and you should see a number of files, including pyethsaletool.py. at this point, open up a command line in this directory. run python pyethsaletool.py, and you should see a list of help instructions. now, let's go over the most common potential issues one by one. 1) i forgot to download my wallet before closing the browser tab. you should receive a backup of your wallet in your email. if you entered a fake email address, and at the same time forgot to download your wallet, then unfortunately you have no recourse. 2) i want to make sure that my ether was actually purchased. run python pyethsaletool.py list -w /path/to/your/wallet.json, substituting the path with the path where you downloaded your wallet to. you should see a record of your purchase. if not, then run python pyethsaletool.py getbtcaddress -w /path/to/your/wallet.json and look up the address on blockchain.info. if there is a nonzero balance, you are in situation #4. 3) i want to make sure that i remember my password. run python pyethsaletool.py getbtcprivkey -w /path/to/your/wallet.json, substituting the path. when it prompts you for the password enter it, and see whether you get an error. if you get an error to do with pkcs7 padding, you entered the wrong password; if you get a btc private key out (ie. a sequence of 51 characters starting with a 5), then you're fine. 4) i sent my btc into the intermediate address, but it never made it to the exodus. run python pyethsaletool.py getbtcprivkey -w /path/to/your/wallet.json, substituting the path appropriately. then, import this private key into the blockchain.info wallet or kryptokit. alternatively, you may also run python pyethsaletool.py finalize -w /path/to/your/wallet.json to finish the purchasing process through python. 5) i want to make sure i will be able to access my ether later. run python pyethsaletool.py getethprivkey -w /path/to/your/wallet.json, substituting the path. then, download pyethereum, install it, and use pyethtool privtoaddr c85ef7d79691fe79573b1a7064c19c1a9819ebdbd1faaab1a8ec92344438aaf4, substituting in the ethereum privkey that you got from the first step. if the address that you get matches the address that you saw when you were purchasing ether, then you know that you have your ethereum private key. 6) i sent more btc into the intermediate address after the web app finalized. this situation is identical to #4. you can recover the btc or finalize it at your leisure. if you have any other issues, please ask them in the comments and they will be added to this article. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements from inside ethereum ðξvhub berlin | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search from inside ethereum ðξvhub berlin posted by gavin wood on december 2, 2014 research & development yesterday was the first proper day of ðξvhub berlin being open, following the first ethereum internal developers' symposium ðξvcon-0. i want to post a few images to let you gauge the mood here. henning, marek, viktor and felix hacking on the couch aeron and brian in discussion alex and jutta conferencing over network and security the rest of the team hacking in the lab previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle [mirror] cantor was wrong: debunking the infinite set hierarchy 2019 apr 01 see all posts this is a mirror of the post at https://medium.com/@vitalikbuterin/cantor-was-wrong-debunking-the-infinite-set-hierarchy-e9ba5015102. by vitalik buterin, phd at university of basel a common strand of mathematics argues that, rather than being one single kind of infinity, there are actually an infinite hierarchy of different levels of infinity. whereas the size of the set of integers is just plain infinite, and the set of rational numbers is just as big as the integers (because you can map every rational number to an integer by interleaving the digits of its numerator and denominator, eg. \(0.456456456.... = \frac{456}{999} = \frac{152}{333} \rightarrow 135323\)), the size of the set of real numbers is some kind of even bigger infinity, because there is no way to make a similar mapping from real numbers to the integers. first of all, i should note that it's relatively easy to see that the claim that there is no mapping is false. here's a simple mapping. for a given real number, give me a (deterministic) python program that will print out digits of it (eg. for π, that might be a program that calculates better and better approximations using the infinite series \(\pi = 4 \frac{4}{3} + \frac{4}{5} \frac{4}{7} + ...\)). i can convert the program into a number (using n = int.from_bytes(open('program.py').read(), 'big')) and then output the number. done. there's the mapping from real numbers to integers. now let's take a look at the most common argument used to claim that no such mapping can exist, namely cantor's diagonal argument. here's an exposition from uc denver; it's short so i'll just screenshot the whole thing: now, here's the fundamental flaw in this argument: decimal expansions of real numbers are not unique. to provide a counterexample in the exact format that the "proof" requires, consider the set (numbers written in binary), with diagonal digits bolded: x[1] = 0.000000... x[2] = 0.011111... x[3] = 0.001111... x[4] = 0.000111... ..... the diagonal gives: 01111..... if we flip every digit, we get the number: \(y =\) 0.10000...... and here lies the problem: just as in decimal, 0.9999.... equals 1, in binary 0.01111..... equals 0.10000..... and so even though the new decimal expansion is not in the original list, the number \(y\) is exactly the same as the number \(x[2]\). note that this directly implies that the halting problem is in fact solvable. to see why, imagine a computer program that someone claims will not halt. let c[1] be the state of the program after one step, c[2] after two steps, etc. let x[1], x[2], x[3].... be a full enumeration of all real numbers (which exists, as we proved above), expressed in base \(2^d\) where \(d\) is the size of the program's memory, so a program state can always be represented as a single "digit". let y = 0.c[1]c[2]c[3]........ this number is by assumption part of the list, so it is one of the x[i] values, and hence it can be computed in some finite amount of time. this has implications in a number of industries, particularly in proving that "turing-complete" blockchains are in fact secure. patent on this research is pending. gav’s ethereum ðξv update ii | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search gav’s ethereum ðξv update ii posted by gavin wood on november 1, 2014 research & development well... what a busy two weeks. i thought it about time to make another update for any of you who might be interested in how we're doing. if you don't already know, i'm gavin, a founder of ethereum and one of the three directors (alongside vitalik and jeffrey) of ethereum ðξv, the development entity building ethereum and all the associated technology. after doing some recruitment on behalf of dev in bucharest with the help of mihai alisie and the lovely roxanna sureanu i spent the last week at my home (and coincidentally, the ethereum hq) in zug, switzerland. during this time i was able to get going on the first prototype of whisper, our secure identity-based communications protocol, finishing with a small irc-like ðapp demonstrating how easy it is to use. for those interested, there is more information on whisper in the ethereum github wiki and a nice little screenshot on my twitter feed. in addition to this i've been helping finalise the soon-to-be-announced poc-7 specification and working towards a poc-8 (final). finally, during our brief time together in zug, jeffrey, vitalik and i drafted our strategy concerning identity and key management; this will be developed further during the coming weeks. ðξvhub berlin in berlin, sarah has been super-busy with the builders getting the hub ready. here's a couple pictures of the work in progress that is the berlin hub. it might not look like much yet, but we're on target to be moved in by mid-november. i'm particularly happy with sarah's efforts to find a genuine 70's barrista espresso machine (-: i'm excited to announce that christian vömel is joining the team in berlin to be the office manager of ðξvhub berlin. christian has many years experience including having worked in an international environment and has even taught office management! he'll be taking some of the load from our frankly much-overworked company secretary aeron buchanan. the team grows we've finalised a number of new hires over the past couple of weeks: network engineer lefteris karapetsas will be joining the berlin team imminently. having considerable experience with state-of-the-art network traffic analysis and deep-packet inspection systems, he'll be helping audit our network protocols, however (like much of our team) truly multidisciplinary, he'll also be working on natspec, the code name for our natural language formal contract specification system, a cornerstone of our transaction security model. i'm happy to announce that ian meikle, the accomplished videographer who co-authored the impressive "koyaanis-glitchy" ethereum brand video has been moved to ðξv to help with the communications team. he who shall be known only as texture has also joined the comms side with stephan to help with the strategy stateside and coordinate the worldwide meetup and hackathon network. great to see such a capable and passionate designer on the team; i know he has a good few ideas for ðapps! two more hires under stephan in the comms team include ken kappler, handling the developer education direction, hackathons, ethereum curriculum and university partnerships. george hallam has also been employed to evangelize ethereum to startups and partners, boost the reach of our formal network and generally help stephan in the quest of having everybody know what ethereum is and how it can help them. jeff's team has also been expanded recently too; he'll be telling you about his developments in an imminent post. further developments aside from the aforementioned progress with whisper and poc-7, christoph has been continuing his great work with the tests repository. christian has been making great progress with the solidity language having recently placed the first solidity-compiled program onto the testnet block chain only a few days ago. marek has studiously been moving c++ over to a json-rpc and javascript front-end fundamentally unified and bound to the go client. alex meanwhile has been grappling with the c++ crypto back-end and has done a great job of reducing bloat and extraneous dependencies. of late, the comms team has some good news brewing, in particular, it is in contact with some world-class education establishments regarding the possibility of eduction partnership and the formation of a network of chapters both in the uk and internationally. watch this space (-: finally, our polish partners at imapp (paweł and artur) have completed their first implementation of the jit-compiled llvm-based evm implementation. they are reporting an average of 30x speedup (as high as 100x!) for non-external evm instructions over the already best-in-class basic c++-based evm implementation. brilliant work and we're looking forward to more improvements and optimisations yet. and the rest... so much to come; there are a couple of announcements (including a slew of imminent hires) i'd love to make but they need to be finalised before i can write about them here. look out for the next update! gav (gavin.nospam.wood@ethdev.com). previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements secret sharing daos: the other crypto 2.0 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search secret sharing daos: the other crypto 2.0 posted by vitalik buterin on december 26, 2014 research & development the crypto 2.0 industry has been making strong progress in the past year developing blockchain technology, including the formalization and in some cases realization of proof of stake designs like slasher and dpos, various forms of scalable blockchain algorithms, blockchains using "leader-free consensus" mechanisms derived from traditional byzantine fault tolerance theory, as well as economic ingredients like schelling consensus schemes and stable currencies. all of these technologies remedy key deficiencies of the blockchain design with respect to centralized servers: scalability knocks down size limits and transaction costs, leader-free consensus reduces many forms of exploitability, stronger pos consensus algorithms reduce consensus costs and improve security, and schelling consensus allows blockchains to be "aware" of real-world data. however, there is one piece of the puzzle that all approaches so far have not yet managed to crack: privacy. currency, dapps and privacy bitcoin brings to its users a rather unique set of tradeoffs with respect to financial privacy. although bitcoin does a substantially better job than any system that came before it at protecting the physical identities behind each of its accounts better than fiat and banking infrastructure because it requires no identity registration, and better than cash because it can be combined with tor to completely hide physical location, the presence of the bitcoin blockchain means that the actual transactions made by the accounts are more public than ever neither the us government, nor china, nor the thirteen year old hacker down the street even need so much as a warrant in order to determine exactly which account sent how much btc to which destination at what particular time. in general, these two forces pull bitcoin in opposite directions, and it is not entirely clear which one dominates. with ethereum, the situation is similar in theory, but in practice it is rather different. bitcoin is a blockchain intended for currency, and currency is inherently a very fungible thing. there exist techniques like merge avoidance which allow users to essentially pretend to be 100 separate accounts, with their wallet managing the separation in the background. coinjoin can be used to "mix" funds in a decentralized way, and centralized mixers are a good option too especially if one chains many of them together. ethereum, on the other hand, is intended to store intermediate state of any kind of processes or relationships, and unfortunately it is the case that many processes or relationships that are substantially more complex than money are inherently "account-based", and large costs would be incurred by trying to obfuscate one's activities via multiple accounts. hence, ethereum, as it stands today, will in many cases inherit the transparency side of blockchain technology much more so than the privacy side (although those interested in using ethereum for currency can certainly build higher-privacy cash protocols inside of subcurrencies). now, the question is, what if there are cases where people really want privacy, but a diaspora-style self-hosting-based solution or a zerocash-style zero-knowledge-proof strategy is for whatever reason impossible for example, because we want to perform calculations that involve aggregating multiple users' private data? even if we solve scalability and blockchain data assets, will the lack of privacy inherent to blockchains mean that we simply have to go back to trusting centralized servers? or can we come up with a protocol that offers the best of both worlds: a blockchain-like system which offers decentralized control not just over the right to update the state, but even over the right to access the information at all? as it turns out, such a system is well within the realm of possibility, and was even conceptualized by nick szabo in 1998 under the moniker of "god protocols" (though, as nick szabo pointed out, we should not use that term for the protocols that we are about to describe here as god is generally assumed or even defined to be pareto-superior to everything else and as we'll soon see these protocols are very far from that); but now with the advent of bitcoin-style cryptoeconomic technology the development of such a protocol may for the first time actually be viable. what is this protocol? to give it a reasonably technically accurate but still understandable term, we'll call it a "secret sharing dao". fundamentals: secret sharing to skip the fun technical details and go straight to applications, click here secret computation networks rely on two fundamental primitives to store information in a decentralized way. the first is secret sharing. secret sharing essentially allows data to be stored in a decentralized way across n parties such that any k parties can work together to reconstruct the data, but k-1 parties cannot recover any information at all. n and k can be set to any values desired; all it takes is a few simple parameter tweaks in the algorithm. the simplest way to mathematically describe secret sharing is as follows. we know that two points make a line: so, to implement 2-of-n secret sharing, we take our secret s, generate a random slope m, and create the line y = mx + s. we then give the n parties the points on the line (1, m + s), (2, 2m + s), (3, 3m + s), etc. any two of them can reconstruct the line and recover the original secret, but one person can do nothing; if you receive the point (4, 12), that could be from the line y = 2x + 4, or y = -10x + 52, or y = 305445x 1221768. to implement 3-of-n secret sharing, we just make a parabola instead, and give people points on the parabola: parabolas have the property that any three points on a parabola can be used to reconstruct the parabola (and no one or two points suffice), so essentially the same process applies. and, more generally, to implement k-of-n secret sharing, we use a degree k-1 polynomial in the same way. there is a set of algorithms for recovering the polynomial from a sufficient set of points in all such cases; they are described in more details in our earlier article on erasure coding. this is how the secret sharing dao will store data. instead of every participating node in the consensus storing a copy of the full system state, every participating node in the consensus will store a set of shares of the state points on polynomials, one point on a different polynomial for each variable that makes up part of the state. fundamentals: computation now, how does the secret sharing dao do computation? for this, we use a set of algorithms called secure multiparty computation (smpc). the basic principle behind smpc is that there exist ways to take data which is split among n parties using secret sharing, perform computations on it in a decentralized way, and end up with the result secret-shared between the parties, all without ever reconstituting any of the data on a single device. smpc with addition is easy. to see how, let's go back to the two-points-make-a-line example, but now let's have two lines: suppose that the x=1 point of both lines a and b is stored by computer p[1], the x=2 point is stored by computer p[2], etc. now, suppose that p[1] computes a new value, c(1) = a(1) + b(1), and b computes c(2) = a(2) + b(2). now, let's draw a line through those two points: so we have a new line, c, such that c = a + b at points x=1 and x=2. however, the interesting thing is, this new line is actually equal to a + b on every point: thus, we have a rule: sums of secret shares (at the same x coordinate) are secret shares of the sum. using this principle (which also applies to higher dimensions), we can convert secret shares of a and secret shares of b into secret shares of a+b, all without ever reconstituting a and b themselves. multiplication by a known constant value works the same way: k times the ith secret share of a is equal to the ith secret share of a*k. multiplication of two secret shared values, unfortunately, is much more involved. the approach will take several steps to explain, and because it is fairly complicated in any case it's worth simply doing for arbitrary polynomials right away. here's the magic. first, suppose that there exist values a and b, secret shared among parties p[1] ... p[n], where a[i] represents the ith share of a (and same for b[i] and b). we start off like this: now, one option that you might think of is, if we can just make a new polynomial c = a + b by having every party store c[i] = a[i] + b[i], can't we do the same for multiplication as well? the answer is, surprisingly, yes, but with a serious problem: the new polynomial has a degree twice as large as the original. for example, if the original polynomials were y = x + 5 and y = 2x 3, the product would be y = 2x^2 + 7x 15. hence, if we do multiplication more than once, the polynomial would become too big for the group of n to store. to avoid this problem, we perform a sort of rebasing protocol where we convert the shares of the larger polynomial into shares of a polynomial of the original degree. the way it works is as follows. first, party p[i] generates a new random polynomial, of the same degree as a and b, which evaluates to c[i] = a[i]*b[i] at zero, and distributes points along that polynomial (ie. shares of c[i]) to all parties. thus, p[j] now has c[i][j] for all i. given this, p[j] calculates c[j], and so everyone has secret shares of c, on a polynomial with the same degree as a and b. to do this, we used a clever trick of secret sharing: because the secret sharing math itself involves nothing more than additions and multiplications by known constants, the two layers of secret sharing are commutative: if we apply secret sharing layer a and then layer b, then we can take layer a off first and still be protected by layer b. this allows us to move from a higher-degree polynomial to a lower degree polynomial but avoid revealing the values in the middle instead, the middle step involved both layers being applied at the same time. with addition and multiplication over 0 and 1, we have the ability to run arbitrary circuits inside of the smpc mechanism. we can define: and(a, b) = a * b or(a, b) = a + b a * b xor(a, b) = a + b 2 * a * b not(a) = 1 a hence, we can run whatever programs we want, although with one key limitation: we can't do secret conditional branching. that is, if we had a computation if (x == 5) <do a> else <do b> then the nodes would need to know whether they are computing branch a or branch b, so we would need to reveal x midway through. there are two ways around this problem. first, we can use multiplication as a "poor man's if" replace something like if (x == 5) <y = 7> with y = (x == 5) * 7 + (x != 5) * y, using either circuits or clever protocols that implement equality checking through repeated multiplication (eg. if we are in a finite field we can check if a == b by using fermat's little theorem on a-b). second, as we will see, if we implement if statements inside the evm, and run the evm inside smpc, then we can resolve the problem, leaking only the information of how many steps the evm took before computation exited (and if we really care, we can reduce the information leakage further, eg. round the number of steps to the nearest power of two, at some cost to efficiency). the secret-sharing based protocol described above is only one way to do relatively simply smpc; there are other approaches, and to achieve security there is also a need to add a verifiable secret sharing layer on top, but that is beyond the scope of this article the above description is simply meant to show how a minimal implementation is possible. building a currency now that we have a rough idea of how smpc works, how would we use it to build a decentralized currency engine? the general way that a blockchain is usually described in this blog is as a system that maintains a state, s, accepts transactions, agrees on which transactions should be processed at a given time and computes a state transition function apply(s, tx) -> s' or invalid. here, we will say that all transactions are valid, and if a transaction tx is invalid then we simply have apply(s, tx) = s. now, since the blockchain is not transparent, we might expect the need for two kinds of transactions that users can send into the smpc: get requests, asking for some specific information about an account in the current state, and update requests, containing transactions to apply onto the state. we'll implement the rule that each account can only ask for balance and nonce information about itself, and can withdraw only from itself. we define the two types of requests as follows: send: [from_pubkey, from_id, to, value, nonce, sig] get: [from_pubkey, from_id, sig] the database is stored among the n nodes in the following format: essentially, the database is stored as a set of 3-tuples representing accounts, where each 3-tuple stores the owning pubkey, nonce and balance. to send a request, a node constructs the transaction, splits it off into secret shares, generates a random request id and attaches the id and a small amount of proof of work to each share. the proof of work is there because some anti-spam mechanism is necessary, and because account balances are private there is no way if the sending account has enough funds to pay a transaction fee. the nodes then independently verify the shares of the signature against the share of the public key supplied in the transaction (there are signature algorithms that allow you to do this kind of per-share verification; schnorr signatures are one major category). if a given node sees an invalid share (due to proof of work or the signature), it rejects it; otherwise, it accepts it. transactions that are accepted are not processed immediately, much like in a blockchain architecture; at first, they are kept in a memory pool. at the end of every 12 seconds, we use some consensus algorithm it could be something simple, like a random node from the n deciding as a dictator, or an advanced neo-bft algorithm like that used by pebble to agree on which set of request ids to process and in which order (for simplicity, simple alphabetical order will probably suffice). now, to fufill a get request, the smpc will compute and reconstitute the output of the following computation: owner_pubkey = r[0] * (from_id == 0) + r[3] * (from_id == 1) + ... + r[3*n] * (from_id == n) valid = (owner_pubkey == from_pubkey) output = valid * (r[2] * (from_id == 0) + r[5] * (from_id == 1) + ... + r[3n + 2] * (from_id == n)) so what does this formula do? it consists of three stages. first, we extract the owner pubkey of the account that the request is trying to get the balance of. because the computation is done inside of an smpc, and so no node actually knows what database index to access, we do this by simply taking all the database indices, multiplying the irrelevant ones by zero and taking the sum. then, we check if the request is trying to get data from an account which is actually owns (remember that we checked the validity of from_pubkey against the signature in the first step, so here we just need to check the account id against the from_pubkey). finally, we use the same database getting primitive to get the balance, and multiply the balance by the validity to get the result (ie. invalid requests return a balance of 0, valid ones return the actual balance). now, let's look at the execution of a send. first, we compute the validity predicate, consisting of checking that (1) the public key of the targeted account is correct, (2) the nonce is correct, and (3) the account has enough funds to send. note that to do this we once again need to use the "multiply by an equality check and add" protocol, but for brevity we will abbreviate r[0] * (x == 0) + r[3] * (x == 1) + ... with r[x * 3]. valid = (r[from_id * 3] == from_pubkey) * (r[from_id * 3 + 1] == nonce) * (r[from_id * 3 + 2] >= value) we then do: r[from_id * 3 + 2] -= value * valid r[from_id * 3 + 1] += valid r[to * 3 + 2] += value * valid for updating the database, r[x * 3] += y expands to the set of instructions r[0] += y * (x == 0), r[3] += y * (x == 1) .... note that all of these can be parallelized. also, note that to implement balance checking we used the >= operator. this is once again trivial using boolean logic gates, but even if we use a finite field for efficiency there do exist some clever tricks for performing the check using nothing but additions and multiplications. in all of the above we saw two fundamental limitations in efficiency in the smpc architecture. first, reading and writing to a database has an o(n) cost as you pretty much have to read and write every cell. doing anything less would mean exposing to individual nodes which subset of the database a read or write was from, opening up the possibility of statistical memory leaks. second, every multiplication requires a network message, so the fundamental bottleneck here is not computation or memory but latency. because of this, we can already see that secret sharing networks are unfortunately not god protocols; they can do business logic just fine, but they will never be able to do anything more complicated even crypto verifications, with the exception of a select few crypto verifications specifically tailored to the platform, are in many cases too expensive. from currency to evm now, the next problem is, how do we go from this simple toy currency to a generic evm processor? well, let us examine the code for the virtual machine inside a single transaction environment. a simplified version of the function looks roughly as follows: def run_evm(block, tx, msg, code): pc = 0 gas = msg.gas stack = [] stack_size = 0 exit = 0 while 1: op = code[pc] gas -= 1 if gas < 0 or stack_size < get_stack_req(op): exit = 1 if op == add: x = stack[stack_size] y = stack[stack_size 1] stack[stack_size 1] = x + y stack_size -= 1 if op == sub: x = stack[stack_size] y = stack[stack_size 1] stack[stack_size 1] = x y stack_size -= 1 ... if op == jump: pc = stack[stack_size] stack_size -= 1 ... the variables involved are: the code the stack the memory the account state the program counter hence, we can simply store these as records, and for every computational step run a function similar to the following: op = code[pc] * alive + 256 * (1 alive) gas -= 1 stack_p1[0] = 0 stack_p0[0] = 0 stack_n1[0] = stack[stack_size] + stack[stack_size 1] stack_sz[0] = stack_size 1 new_pc[0] = pc + 1 stack_p1[1] = 0 stack_p0[1] = 0 stack_n1[1] = stack[stack_size] stack[stack_size 1] stack_sz[1] = stack_size 1 new_pc[1] = pc + 1 ... stack_p1[86] = 0 stack_p0[86] = 0 stack_n1[86] = stack[stack_size 1] stack_sz[86] = stack_size 1 new_pc[86] = stack[stack_size] ... stack_p1[256] = 0 stack_p0[256] = 0 stack_n1[256] = 0 stack_sz[256] = 0 new_pc[256] = 0 pc = new_pc[op] stack[stack_size + 1] = stack_p1[op] stack[stack_size] = stack_p0[op] stack[stack_size 1] = stack_n1[op] stack_size = stack_sz[op] pc = new_pc[op] alive *= (gas < 0) * (stack_size < 0) essentially, we compute the result of every single opcode in parallel, and then pick the correct one to update the state. the alive variable starts off at 1, and if the alive variable at any point switches to zero, then all operations from that point simply do nothing. this seems horrendously inefficient, and it is, but remember: the bottleneck is not computation time but latency. everything above can be parallelized. in fact, the astute reader may even notice that the entire process of running every opcode in parallel has only o(n) complexity in the number of opcodes (particularly if you pre-grab the top few items of the stack into specified variables for input as well as output, which we did not do for brevity), so it is not even the most computationally intensive part (if there are more accounts or storage slots than opcodes, which seems likely, the database updates are). at the end of every n steps (or for even less information leakage every power of two of steps) we reconstitute the alive variable and if we see that alive = 0 then we halt. in an evm with many participants, the database will likely be the largest overhead. to mitigate this problem, there are likely clever information leakage tradeoffs that can be made. for example, we already know that most of the time code is read from sequential database indices. hence, one approach might be to store the code as a sequence of large numbers, each large number encoding many opcodes, and then use bit decomposition protocols to read off individual opcodes from a number once we load it. there are also likely many ways to make the virtual machine fundamentally much more efficient; the above is meant, once again, as a proof of concept to show how a secret sharing dao is fundamentally possible, not anything close to an optimal implementation. additionally, we can look into architectures similar to the ones used in scalability 2.0 techniques to highly compartmentalize the state to further increase efficiency. updating the n the smpc mechanism described above assumes an existing n parties involved, and aims to be secure against any minority of them (or in some designs at least any minority less than 1/4 or 1/3) colluding. however, blockchain protocols need to theoretically last forever, and so stagnant economic sets do not work; rather, we need to select the consensus participants using some mechanism like proof of stake. to do this, an example protocol would work as follows: the secret sharing dao's time is divided into "epochs", each perhaps somewhere between an hour and a week long. during the first epoch, the participants are set to be the top n participants during the genesis sale. at the end of an epoch, anyone has the ability to sign up to be one of the participants in the next round by putting down a deposit. n participants are randomly chosen, and revealed. a "decentralized handoff protocol" is carried out, where the n participants simultaneously split their shares among the new n, and each of the new n reconstitutes their share from the pieces that they received essentially, the exact same protocol as was used for multiplication. note that this protocol can also be used to increase or decrease the number of participants. all of the above handles decentralization assuming honest participants; but in a cryptocurrency protocol we also need incentives. to accomplish that, we use a set of primitives called verifiable secret sharing, that allow us to determine whether a given node was acting honestly throughout the secret sharing process. essentially, this process works by doing the secret sharing math in parallel on two different levels: using integers, and using elliptic curve points (other constructions also exist, but because cryptocurrency users are most familiar with the secp256k1 elliptic curve we'll use that). elliptic curve points are convenient because they have a commutative and associative addition operator in essence, they are magic objects which can be added and subtracted much like numbers can. you can convert a number into a point, but not a point into a number, and we have the property that number_to_point(a + b) = number_to_point(a) + number_to_point(b). by doing the secret sharing math on the number level and the elliptic curve point level at the same time, and publicizing the elliptic curve points, it becomes possible to verify malfeasance. for efficiency, we can probably use a schellingcoin-style protocol to allow nodes to punish other nodes that are malfeasant. applications so, what do we have? if the blockchain is a decentralized computer, a secret sharing dao is a decentralized computer with privacy. the secret sharing dao pays dearly for this extra property: a network message is required per multiplication and per database access. as a result, gas costs are likely to be much higher than ethereum proper, limiting the computation to only relatively simple business logic, and barring the use of most kinds of cryptographic calculations. scalability technology may be used to partially offset this weakness, but ultimately there is a limit to how far you can get. hence, this technology will probably not be used for every use case; instead, it will operate more like a special-purpose kernel that will only be employed for specific kinds of decentralized applications. some examples include: medical records keeping the data on a private decentralized platform can potentially open the door for an easy-to-use and secure health information system that keeps patients in control of their data. particularly, note that proprietary diagnosis algorithms could run inside the secret sharing dao, allowing medical diagnosis as a service based on data from separate medical checkup firms without running the risk that they will intentionally or unintentionally expose your private details to insurers, advertisers or other firms. private key escrow a decentralized m-of-n alternative to centralized password recovery; could be used for financial or non-financial applications multisig for anything even systems that do not natively support arbitrary access policies, or even m-of-n multisignature access, now will, since as long as they support cryptography you can stick the private key inside of a secret sharing dao. reputation systems what if reputation scores were stored inside a secret sharing dao so you could privately assign reputation to other users, and have your assignment count towards the total reputation of that user, without anyone being able to see your individual assignments? private financial systems secret sharing daos could provide an alternative route to zerocash-style fully anonymous currency, except that here the functionality could be much more easily extended to decentralized exchange and more complex smart contracts. business users may want to leverage some of the benefits of running their company on top of crypto without necessarily exposing every single one of their internal business processes to the general public. matchmaking algorithms find employers, employees, dating partners, drivers for your next ride on decentralized uber, etc, but doing the matchmaking algorithm computations inside of smpc so that no one sees any information about you unless the algorithm determines that you are a perfect match. essentially, one can think of smpc as offering a set of tools roughly similar to that which it has been theorized would be offered by cryptographically secure code obfuscation, except with one key difference: it actually works on human-practical time scales. further consequences aside from the applications above, what else will secret sharing daos bring? particularly, is there anything to worry about? as it turns out, just like with blockchains themselves, there are a few concerns. the first, and most obvious, issue is that secret sharing daos will substantially increase the scope of applications that can be carried out in a completely private fashion. many advocates of blockchain technology often base a large part of their argument on the key point that while blockchain-based currencies offer an unprecedented amount of anonymity in the sense of not linking addresses to individual identities, they are at the same time the most public form of currency in the world because every transaction is located on a shared ledger. here, however, the first part remains, but the second part disappears completely. what we have left is essentially total anonymity. if it turns out to be the case that this level of anonymity allows for a much higher degree of criminal activity, and the public is not happy with the tradeoff that the technology brings, then we can predict that governments and other institutions in general, perhaps even alongside volunteer vigilante hackers, will try their best to take these systems down, and perhaps they would even be justified. fortunately for these attackers, however, secret sharing daos do have an inevitable backdoor: the 51% attack. if 51% of the maintainers of a secret sharing dao at some particular time decide to collude, then they can uncover any of the data that is under their supervision. furthermore, this power has no statute of limitations: if a set of entities who formed over half of the maintaining set of a secret sharing dao at some point many years ago collude, then even then the group would be able to unearth the information from that point in time. in short, if society is overwhelmingly opposed to something being done inside of a secret sharing dao, there will be plenty of opportunity for the operators to collude to stop or reveal what's going on. a second, and subtler, issue is that the concept of secret sharing daos drives a stake through a cherished fact of cryptoeconomics: that private keys are not securely tradeable. many protocols explicitly, or implicitly, rely on this idea, including non-outsourceable proof of work puzzles, vlad zamfir and pavel kravchenko's proof of custody, economic protocols that use private keys as identities, any kind of economic status that aims to be untradeable, etc. online voting systems often have the requirement that it should be impossible to prove that you voted with a particular key, so as to prevent vote selling; with secret sharing daos, the problem is that now you actually can sell your vote, rather simply: by putting your private key into a contract inside of a secret sharing dao, and renting out access. the consequences of this ability to sell private keys are quite far reaching in fact, they go so far as to almost threaten the security of the strongest available system underlying blockchain security: proof of stake. the potential concern is this: proof of stake derives its security from the fact that users have security deposits on the blockchain, and these deposits can potentially be taken away if the user misacts in some fashion (double-voting, voting for a fork, not voting at all, etc). here, private keys become tradeable, and so security deposits become tradeable as well. we must ask the question: does this compromise proof of stake? fortunately, the answer is no. first of all, there are strong lemon-theoretic arguments for why no one would actually want to sell their deposit. if you have a deposit of $10, to you that's worth $10 minus the tiny probability that you will get hacked. but if you try to sell that deposit to someone else, they will have a deposit which is worth $10, unless you decide to use your private key to double-vote and thus destroy the deposit. hence, from their point of view, there is a constant overhanging risk that you will act to take their deposit away, and you personally have no incentive not to do that. the very fact that you are trying to sell off your deposit should make them suspicious. hence, from their point of view, your deposit might only be worth, say, $8. you have no reason to sacrifice $10 for $8, so as a rational actor you will keep the deposit to yourself. second, if the private key was in the secret sharing dao right from the start, then by transferring access to the key you would personally lose access to it, so you would actually transfer the authority and the liability at the same time from an economic standpoint, the effect on the system would be exactly the same as if one of the deposit holders simply had a change of personality at some point during the process. in fact, secret sharing daos may even improve proof of stake, by providing a more secure platform for users to participate in decentralized stake pools even in protocols like tendermint, which do not natively support such functionality. there are also other reasons why the theoretical attacks that secret sharing daos make possible may in fact fail in practice. to take one example, consider the case of non-outsourceable puzzles, computational problems which try to prove ownership of a private key and a piece of data at the same time. one kind of implementation of a non-outsourceable puzzle, used by permacoin, involves a computation which needs to "bounce" back and forth between the key and the data hundreds of thousands of times. this is easy to do if you have the two pieces of data on the same piece of hardware, but becomes prohibitively slow if the two are separated by a network connection and over a secret sharing dao it would be nearly impossible due to the inefficiencies. as a result, one possible conclusion of all this is that secret sharing daos will lead to the standardization of a signature scheme which requires several hundred millions of rounds of computation preferably with lots and lots of serial multiplication to compute, at which point every computer, phone or internet-of-things microchip would have a built-in asic to do it trivially, secret sharing daos would be left in the dust, and we would all move on with our lives. how far away? so what is left before secret sharing dao technology can go mainstream? in short, quite a bit, but not too much. at first, there is certainly a moderate amount of technical engineering involved, at least on the protocol level. someone needs to formalize an smpc implementation, together with how it would be combined with an evm implementation, probably with many restrictions for efficiency (eg. hash functions inside of smpc are very expensive, so merkle tree storage may disappear in favor of every contract having a finite number of storage slots), a punishment, incentive and consensus framework and a hypercube-style scalability framework, and then release the protocol specification. from that point, it's a few months of development in python (python should be fine, as by far the primary bottleneck will be network latency, not computation), and we'll have a working proof of concept. secret sharing and smpc technology has been out there for many years, and academic cryptographers have been talking about how to build privacy-preserving applications using m-of-n-based primitives and related technologies such as private information retrieval for over a decade. the key contribution made by bitcoin, however, is the idea that m-of-n frameworks in general can be much more easily bootstrapped if we add in an economic layer. a secret sharing dao with a currency built in would provide incentives for individuals to participate in maintaining the network, and would bootstrap it until the point where it could be fully self-sustaining on internal applications. thus, altogether, this technology is quite possible, and not nearly so far away; it is only a matter of time until someone does it. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements on long-term cryptocurrency distribution models | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on long-term cryptocurrency distribution models posted by vitalik buterin on may 24, 2014 research & development one of the challenges when creating a new cryptocurrency is figuring out what the distribution model is going to be. who is going to receive the currency units, at what time, and what is the mechanism that decides? despite the crucial importance of this question, there has actually been comparatively little thought into the issue compared with other aspects of currency, like consensus algorithms and feature sets. the question is particularly challenging because, just like many other problems in the cryptocurrency space that have parallels in the “real world” at large, cryptocurrencies also face the requirement of decentralization: it is considered unacceptable to have a cryptographic platforms whose continued operation depends on the existence of any specific party in the long term. given this rather stringent requirement, how should a new currency distribute itself? so far, the problem is still in its very early stages of discussion. while the question of short-term distribution is a highly dynamic debate between different types of asset carryovers, one-way transfers, two-way pegs, pre-mines, pre-sales and other mechanisms coming out almost every month, long-term distribution in nearly every cryptocurrency now follows one of two strategies: nothing at all, or mining. the reason why having a fixed never-growing supply is undesirable is obvious: it encourages wealth concentration and creates a static community of holders without an effective way for new people to get in, and it means that the coin has no way to incentive any specific kind of activity in the long term. the issue with mining, however, is more subtle. cryptocurrency mining generally serves two functions; first, it provides a way of securing the network, and second, it serves as a distribution model, giving hundreds of thousands of people around the world a way of getting access to a few coins. so far, mining has been considered necessary for the former, and an effective way of doing the latter. more recently, however, there has been a substantial amount of interest and research into proof of stake, including strategies such astransactions as proof-of-stake, delegated proof of stake and a partial solution to nothing-at-stake, slasher, suggesting that mining might not be necessary after all. second, the rise of both asics and professional gpu farms is turning mining itself into an increasingly concentrated and quasi-centralized community, so any new mining-distributed currency will quickly be dominated by professional companies and not “the people” at large. if both trends continue, and mining proves to be a bad model for distribution, it will therefore need to be replaced. but then, the question is, by what? so far, we know of several answers: pretend that the problem does not exist. this is the solution that has been taken by most proof-of-stake cryptocurrencies, and surprisingly enough even proof-of-work currencies, today. centralized distribution: let some central authority hand out coins according to some formula. useful proof-of-work: hand out coins to anyone who performs a particular socially useful computation, eg. weather prediction. this algorithm need not be used for consensus; it can exist simply to distribute coins while proof-of-stake does the hard work of maintaining consensus. algorithmic consensus distribution. essentially, some kind of dynamic, adaptive consensus-based process for determining who gets new coins. the second is theoretically the most powerful; currency units can be distributed either to everyone in the world for maximum fairness or to pay bounties for protocol development, external charitable causes or anything else. however, at the same time actually using such a mechanism arguably kills the whole point of a cryptocurrency: that it is decentralized and depends on no specific party for its continued existence. thus, we can think of the centralized distributor as an ideal that we want to approach, sort of like the ideal of a bureaucrat god found in economic efficiency theory, and see how close to that ideal we can approach while still maintaining a structure that is guaranteed, or at least highly likely, to remain stable in the long term. useful proof of work as distribution: a relaxed algorithm useful proof of work is likely the simpler idea. initially, it was considered impossible to make a proof of work based on useful computation because of the verification problem: a proof-of-work task cannot take longer than a few thousands steps because every node in the network also needs to verify it to accept the block. primecoin was the closest we got, and even there computing chains of prime numbers is not really all that useful. now, thanks to the existence of a programming environment with a built-in computational stack trace mechanism, there is actually an alternative approach that removes this particular obstacle, using spot-checking and deposit sacrifices to make sure that work is being done correctly. the approximate algorithm for doing so is as follows. suppose that f(k) is a function that takes 32 bytes of random data as an input, carries out some computation taking n steps (where n is fairly large, say ten billion) and then returns a value r which is socially useful. in order to perform one round of mining, start off by choosing a random m, and let b be the block header. let k = sha3(b + m) as the seed. define a function step(p, d) -> d' where p is the program code, d is some tuple of data perhaps including stack, memory and program counter representing the state of the computation, and step carries out one computational step and returns the modified computational state d'. let d[0] = { pc: 0, stack: [], memory: [k] } (or some other construction involving k in a different computational model). let d[i] = step(p, d[i-1]) where p is the program corresponding to the evaluation of f. d[n] should, in some appropriate fashion, contain the result of f. define h as a hash function of d[i]; something like sha3(pc + str(stack) + str(memory)) satisfies as a quick-and-dirty option. let h[i] = h(d[i]). compute all d[i] and all h[i] and let r be the root of a merkle tree of all h[i]. if r < 2^256 / d then the work is valid and the miner is entitled to a reward. basically, we take the state of the program after each computational step (we can optionally make step process the execution of a few thousand computational steps for greater efficiency; this does not seriously compromise anything), and build a merkle tree out of the whole thing and look at the root. this is somewhat tricky to implement; fortunately, however, the ethereum virtual machine and block structure is already almost an exact replica of this algorithm, so one could take that code and use it almost verbatim. the algorithm described above by itself has an obvious hole in it: it is not easy-to-verify, so fraudulent miners can easily pollute the network with bad-seeming blocks. thus, as an anti-spam and anti-fraud mechanism, we require the following: to be able to mine, nodes must purchase a “mining bond” of price n * r (say, r = 10^18 and n = 100), which returns to the miner after 10000 blocks. each mining bond allows the miner to submit one work at a time. if a miner submits a seemingly-valid work, including the m and k values, the root, and the socially useful output, then the mining bond reward increases by r anyone else with a mining bond can check the work themselves. if the merkle root at the end is inconsistent, then they can publish a “challenge” transaction consisting of some number (say, 16) of sub-nodes. at that point, the original submitter has the choice of either giving up (as defined by not posting a response within 25 blocks), sacrificing their entire mining bond to the checker, or make a “response” transaction pointing out the first of those subnodes that they disagree with. if a response is submitted, the challenger must respond going down one level further, providing the sixteen subnodes between the last agreed subnode and the first disagreed subnode, and so forth, until the process converges upon the interval between two adjacenth[i] and h[i+1] values in the tree. at that point, the miner must submit the values of d[i] and d[i+1] in a transaction, which is considered valid if and only if p(d[i]) = d[i+1]. the problem is, however, that the process of checking takes as long as the original computation itself, so there does need to be an explanation as to why anyone would do it. if all miners attempt to cheat frequently, then it makes sense to perform spot-checks in order to collect the deposit (which we assumed to be 100x), but if miners realize this and as a result don’t cheat then there is no longer an incentive to check, so no one would check and miners would have free rein to cheat. this is a classichawk-dove equilibrium paradox, and can be solved by game theory (here, we assume that mining has a cost of 0.5 and a reward of 1): cheats does not cheat checks (-100, 101) (0.5,-0.5) does not check (1,0) (0.5,0) computing a mixed-strategy equilibrium in this simplified two-player model shows the miner cheating 0.5% of the time and the checker checking 0.5% of the time; under those two conditions, each player is indifferent to the strategy of the other so there is no opportunity for either one to further optimize and cheat. if we push closer to the economic equilibrium of mining and we say that mining has a cost of 0.9, then the equilibrium has a cheating rate of 0.9% and a checking rate of 0.9%. thus, economically driven spot-checking is a legitimate strategy for ratting out fraudulent mining attempts, and can keep cheating rates arbitrarily low if we are willing to push up collateral requirements. so what kind of work can we do? first of all, it might be better not to include computation that is incapable of handling noise, ie. where a bad answer accepted as a good answer does more than 100x as much bad as an actual good answer. second, the algorithm here allows for work that is not easy-to-verify, but it does nothing to allow work that is data-heavy. for example, seti is data-heavy – you need to have a picture of the sky in order to search it for aliens. third, the algorithm must be parallelization-friendly. running a machine learning algorithm on terabytes of data is not really something that can be split into discrete chunks, even large-sized ones. the second criterion can potentially be relaxed; because there isn’t really any benefit to mining with bad data versus good data, an seti foundation can be set up which provides a stream of data for miners to work with, and adds a very small subsidy to encourage miners to use it. theoretically, the foundation can even be decentralized and run as a proof-of-stake-voting algorithm on a blockchain. the simplest kind of socially useful computation to use, however, might be genetic algorithms. genetic algorithms are often used to find solutions to problems that are intractable in closed-form, like finding optimal radio antenna shapes, spaceflight trajectories, aerodynamic shapes, and so forth; the blockchain may provide an ideal environment for doing such computation on everyone’s nodes for free. certain classes of data search and aggregation puzzles could also potentially be split up, though they are much more data-heavy whereas genetic algorithms are close to data-free once launched. parliaments and better algorithms algorithmic consensus distribution is the more interesting possibility. what if there can be a consensus algorithm to distribute tokens over time, where that algorithm can reward arbitrary good work? for example, one might want to pay bounties to people who contribute to the ecosystem, or even to the world in general. the simplest approach here seems to be to randomly select a “parliament” – every n blocks, stakeholders can vote on 200 nodes that will make the decision of where the newly generated funds will go. the obvious question to ask is: what are the economics of this? in theory, the nodes will want to select the distribution that optimally benefits the community as a whole, so as to maximize their chance of getting re-elected. however, are there opportunities for corruption? we all know that traditional democracy is highly imperfect, so how do we know that our crypto-enabled wealth distribution scheme will be any better? fortunately, there is one strong argument to be made that it actually will be. the reason is that traditional democracies have a number of very serious failure modes; for example, a parliament can seize people’s property, conscript people into armies for war, restrict free speech, etc. in this case, however, there is a very clear and obvious upper bound on how much damage a parliament could do: it could redirect the money to split among itself. there is also the risk that the parliament will crowdfund something which is a public bad to society, but a public good among themselves (eg. a war), but they have no existing military apparatus to latch onto and no existing public consensus that they are supposed to be using coercive power for any reason at all so they are in no better a position to do such a thing than any other group commanding a similar level of economic resources. thus, if we suppose that parliaments fail, say, 33% of the time, then we can see how in a democracy this would be catastrophic but here it only means that the distribution mechanism becomes 67% as useful as it could be. another criticism is that such a mechanism, no matter how it may be constructed, will invariably create some sort of political governance class, and thus will stabilize around a particular small set of political viewpoints, generate its own form of inequality, and eventually lead to a long-term hostile takeover. this would be limited in effect, but even still at its worst 100% of the new currency issuance will be siphoned off by a crypto-political elite. one solution is to make parliaments randomly selected (ie. demarchy) rather than elected, reducing the chance of such conspiracies further but at the cost of weakening the parliament’s expected level of expertise on optimal distribution and its ability to form long-term consistent institutions; however, if we want to create a system that has the political image of being neutral and decentralized that is perhaps something that we actually want. however, we probably can, and certainly must at least try, to be more imaginative. parliaments and voting are only the simplest and crudest form of having a decentralized organization; there are almost certainly better alternatives based on principles such as holarchy, liquid democracy, futarchy and various combinations of these and other ideas that we have not thought of but that will become possible because of the much higher degree of both interconnectedness and information processing efficiency provided by modern technology. ideally, as much of the process as possible would be in some fashion automated – the process should function as a dao, not a do, and the position of highest power, or the closest philosophical analog of such a thing, should be held by an algorithm and not a set of people – perhaps a sacrifice from the point of view of optimality at any particular time, but, one might argue, a boon for long-term stability, and an especially appropriate choice for a cryptographic platform that intends to claim some concept of neutrality. a simple futarchy-based implementation might work as follows. suppose that there are n projects asking for a grant consisting of the entire currency supply to be distributed during some time period, and the desire is to select the one that will maximize the value of the coin after one year. we create n sub-tokens, t[0] ... t[n-1], where the value of t[i] is zero if project i does not get chosen but can be redeemed for one currency unit after one year if the project does get chosen. then, we create subtokens r[0] ... r[n-1], where the value of r[i] is zero if the project does not get chosen or an amount of currency units equal to 232 computational steps in value (we include a small useful-pow or useless-pow market into the coin for this purpose) if the project does get chosen. now, suppose that the probability of project i getting chosen is p[i] and the value of the token in the event that project i gets chosen after one year is v[i]. we note that the value of t[i] is p[i] _ v[i] and the value of r[i] is p[i] _ k where k is the cost of computing 232 computational steps. hence, the project with maximump[i] / r[i] also maximizes v[i] / k and hence v[i], so that project is assumed to maximize the value of the coin and hence chosen. the only challenge left is figuring out what the risks of market manipulation attacks are assuming there are individual parties with non-negligible market power. this strategy seems more mathematically clean and less vulnerable to turning into something centralized, but on the other hand there seem to be fewer safeguards to prevent it from becoming evil. the best response might simply be that a coin run by an evil dao will lose public support, and hence will lose value, so the futarchy algorithm itself might select against such undesirable actions. second, of course, the futarchy does not command a military and there is no pre-existing public consensus that it is entitled to employ any kind of coercion. ultimately, both of these approaches could be combined. one can have a parliament, or a futarchy, select useful proof of work algorithms or even data for specific useful proof of work algorithms, or one can have a parliament or futarchy with useful proof of work as its voting mechanism. however, one important conclusion here is that both of the algorithms described are complicated; there is no easy solution to figuring out how to distribute coins in a good way. which, given the state of the financial system at large, makes sense; if it was easy to distribute coins fairly then the us dollar and other fiat currencies would have likely been overthrown in favor of such alternatives in at least some parts of the world a long time ago. because of the complexity involved, it is unlikely that either of these will be used for ether itself; ether is intended to be boring crypto-gasoline with simple properties to target maximum stability and reliability, not a super-advanced economically innovative decentralized autonomous organization. so if you want to see geneticalgocoin, futarchycoin and parliamentcoin developed, feel free to run them on top of ethereum as sub-currencies; the serpent compiler is all yours to play with. credit to neal koblitz for suggesting the idea of spot-checking and convincing me of the importance of useful pow, robin hanson for inventing futarchy, and realistically probably at least several cryptographers who came up with the concept of multi-round challenge-response protocols before me previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the latest evm: “ethereum is a trust-free closure system” | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the latest evm: “ethereum is a trust-free closure system” posted by vitalik buterin on march 20, 2014 research & development in the past two weeks our lead c++ developer, gavin wood, and myself have been spending a lot of time meeting the local ethereum community in san francisco and silicon valley. we were very excited to see such a large amount of interest in our project, and the fact that after only two months we have a meetup group that comes together every week, just like the bitcoin meetup, with over thirty people attending each time. people in the community are taking it upon themselves to make educational videos, organize events and experiment with contracts, and one person is even independently starting to write an implementation of ethereum in node.js. at the same time, however, we had the chance to take another look at the ethereum protocols, see where things are still imperfect, and agree on a large array of changes that will be integrated, likely with only minimal modification, into the poc 3.5 clients. transactions as closures in es1 and es2, the mktx opcode, which allowed contracts to send transactions triggering other contracts, had one very non-intuitive feature: although one would naturally expect mktx to be like a function call, processing the entire transaction immediately and then continuing on with the rest of the code, in reality mktx did not work this way. instead, the execution of the call is deferred toward the end – when mktx was called, a new transaction would be pushed to the front of the transaction stack of the block, and when the execution of the first transaction ends the execution of the second transaction begins. for example, this is something that you might expect to work: x = array() x[0] = "george" x[1] = mypubkey mktx(namecoin,10^20,x,2) if contract.storage(namecoin)["george"] == mypubkey: registration_successful = 1 else: registration_successful = 0 // do more stuff... use the namecoin contract to try to register “george”, then use the extro opcode to see if the registration is successful. this seems like it should work. however, of course, it doesn’t. in evm3 (no longer es3), we fix this problem. we do this by taking an idea from es2 – creating a concept of reusable code, functions and software libraries, and an idea from es1 – keeping it simple by keeping code as a sequential set of instructions in the state, and merging the two together into a concept of “message calls”. a message call is an operation executed from inside a contract which takes a destination address, an ether value, and some data as input and calls the contract with that ether value and data, but which also, unlike a transaction, returns data as an output. there is thus also a new return opcode which allows contract execution to return data. with this system, contracts can now be much more powerful. contracts of the traditional sort, performing certain data upon receiving message calls, can still exist. but now, however, two other design patterns also become possible. first, one can now create a proprietary data feed contract; for example, bloomberg can publish a contract into which they push various asset prices and other market data, and include in its contract an api that returns the internal data as long as the incoming message call sends at least 1 finney along with it. the fee can’t go too high; otherwise contracts that fetch data from the bloomberg contract once per block and then provide a cheaper passthrough will be profitable. however, even with fees equal to the value of perhaps a quarter of a transaction fee, such a data-feeding business may end up being very viable. the extro opcode is removed to facilitate this functionality, ie. contracts are now opaque from inside the system, although from the outside one can obviously simply look at the merkle tree. second, it is possible to create contracts that represent functions; for example, one can have a sha256 contract or an ecmul contract to compute those respective functions. there is one problem with this: twenty bytes to store the address to call a particular function might be a bit much. however, this can be solved by creating one “stdlib” contract which contains a few hundred clauses for common functions, and contracts can store the address of this contract once as a variable and then access it many times simply as “x” (technically, “push 0 mload”). this is the evm3 way of integrating the other major idea from es2, the concept of standard libraries. ether and gas another important change is this: contracts no longer pay for contract execution, transactions do. when you send a transaction, you now need to include a basefee and a maximum number of steps that you’re willing to pay for. at the start of transaction execution, the basefee multiplied by the maxsteps is immediately subtracted from your balance. a new counter is then instantiated, called gas, that starts off with the number of steps that you have left. then, transaction execution starts as before. every step costs 1 gas, and execution continues until either it naturally halts, at which point all remaining gas times the provided basefee is returned to the sender, or the execution runs out of gas; in that case, all execution is reverted but the entire fee is still paid. this approach has two important benefits. first, it allows miners to know ahead of time the maximum quantity of gas that a transaction will consume. second, and much more importantly, it allows contract writers to spend much less time focusing on making the contract “defensible” against dummy transactions that try to sabotage the contract by forcing it to pay fees. for example, consider the old 5-line namecoin: if tx.value < block.basefee * 200: stop if !contract.storage[tx.data[0]] or tx.data[0] = 100: contract.storage[tx.data[0]] = tx.data[1] two lines, no checks. much simpler. focus on the logic, not the protocol details. the main weakness of the approach is that it means that, if you send a transaction to a contract, you need to precalculate how long the execution will take (or at least set a reasonable upper bound you’re willing to pay), and the contract has the power to get into an infinite loop, use up all the gas, and force you to pay your fee with no effect. however, this is arguably a non-issue; when you send a transaction to someone, you are already implicitly trusting them not to throw the money into a ditch (or at least not complain if they do), and it’s up to the contract to be reasonable. contracts may even choose to include a flag stating how much gas they expect to require (i hereby nominate prepending “push 4 jmp ” to execution code as a voluntary standard) there is one important extension to this idea, which applies to the concept of message calls: when a contract makes a message call, the contract also specifies the amount of gas that the contract on the other end of the call has to use. just as at the top level, the receiving contract can either finish execution in time or it can run out of gas, at which point execution reverts to the start of the call but the gas is still consumed. alternatively, contracts can put a zero in the gas fields; in that case, they are trusting the sub-contract with all remaining gas. the main reason why this is necessary is to allow automatic contracts and human-controlled contracts to interact with each other; if only the option of calling a contract with all remaining gas was available, then automatic contracts would not be able to use any human-controlled contracts without absolutely trusting their owners. this would make m-of-n data feed applications essentially nonviable. on the other hand, this does introduce the weakness that the execution engine will need to include the ability to revert to certain previous points (specifically, the start of a message call). the new terminology guide with all of the new concepts that we have introduced, we have standardized on a few new terms that we will use; hopefully, this will help clear up discussion on the various topics. external actor: a person or other entity able to interface to an ethereum node, but external to the world of ethereum. it can interact with ethereum through depositing signed transactions and inspecting the block-chain and associated state. has one (or more) intrinsic accounts. address: a 160-bit code used for identifying accounts. account: accounts have an intrinsic balance and transaction count maintained as part of the ethereum state. they are owned either by external actors or intrinsically (as an indentity) an autonomous object within ethereum. if an account identifies an autonomous object, then ethereum will also maintain a storage state particular to that account. each account has a single address that identifies it. transaction: a piece of data, signed by an external actor. it represents either a message or a new autonomous object. transactions are recorded into each block of the block-chain. autonomous object: a virtual object existant only within the hypothetical state of ethereum. has an intrinsic address. incorporated only as the state of the storage component of the vm. storage state: the information particular to a given autonomous object that is maintained between the times that it runs. message: data (as a set of bytes) and value (specified as ether) that is passed between two accounts in a perfectly trusted way, either through the deterministic operation of an autonomous object or the cryptographically secure signature of the transaction. message call: the act of passing a message from one account to another. if the destination account is an autonomous object, then the vm will be started with the state of said object and the message acted upon. if the message sender is an autonomous object, then the call passes any data returned from the vm operation. gas: the fundamental network cost unit. paid for exclusively by ether (as of poc-3.5), which is converted freely to and from gas as required. gas does not exist outside of the internal ethereum computation engine; its price is set by the transaction and miners are free to ignore transactions whose gas price is too low. long term view soon, we will release a full formal spec of the above changes, including a new version of the whitepaper that takes into account all of these modifications, as well as a new version of the client that implements it. later on, further changes to the evm will likely be made, but the eth-hll will be changed as little as possible; thus, it is perfectly safe to write contracts in eth-hll now and they will continue to work even if the language changes. we still do not have a final idea of how we will deal with mandatory fees; the current stop-gap approach is now to have a block limit of 1000000 operations (ie. gas spent) per block. economically, a mandatory fee and a mandatory block limit are essentially equivalent; however, the block limit is somewhat more generic and theoretically allows a limited number of transactions to get in for free. there will be a blog post covering our latest thoughts on the fee issue shortly. the other idea that i had, stack traces, may also be implemented later. in the long term, maybe even beyond ethereum 1.0, perhaps the holy grail is attack the last two “intrinsic” parts of the system, and see if we can turn them too into contracts: ether and ecdsa. in such a system, ether would still be the privileged currency in the system; the current thinking is that we will premine the ether contract into the index “1″ so it takes nineteen fewer bytes to use it. however, the execution engine would become simpler since there would no longer be any concept of a currency – instead, it would all be about contracts and message calls. another interesting benefit is that this would allow ether and ecdsa to be decoupled, making ether optionally quantum-proof; if you want, you could make an ether account using an ntru or lamport contract instead. a detriment, however, is that proof of stake would not be possible without a currency that is intrinsic at the protocol level; that may be a good reason not to go in this direction. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements state tree pruning | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search state tree pruning posted by vitalik buterin on june 26, 2015 research & development one of the important issues that has been brought up over the course of the olympic stress-net release is the large amount of data that clients are required to store; over little more than three months of operation, and particularly during the last month, the amount of data in each ethereum client's blockchain folder has ballooned to an impressive 10-40 gigabytes, depending on which client you are using and whether or not compression is enabled. although it is important to note that this is indeed a stress test scenario where users are incentivized to dump transactions on the blockchain paying only the free test-ether as a transaction fee, and transaction throughput levels are thus several times higher than bitcoin, it is nevertheless a legitimate concern for users, who in many cases do not have hundreds of gigabytes to spare on storing other people's transaction histories. first of all, let us begin by exploring why the current ethereum client database is so large. ethereum, unlike bitcoin, has the property that every block contains something called the "state root": the root hash of a specialized kind of merkle tree which stores the entire state of the system: all account balances, contract storage, contract code and account nonces are inside. the purpose of this is simple: it allows a node given only the last block, together with some assurance that the last block actually is the most recent block, to "synchronize" with the blockchain extremely quickly without processing any historical transactions, by simply downloading the rest of the tree from nodes in the network (the proposed hashlookup wire protocol message will faciliate this), verifying that the tree is correct by checking that all of the hashes match up, and then proceeding from there. in a fully decentralized context, this will likely be done through an advanced version of bitcoin's headers-first-verification strategy, which will look roughly as follows: download as many block headers as the client can get its hands on. determine the header which is on the end of the longest chain. starting from that header, go back 100 blocks for safety, and call the block at that position p100(h) ("the hundredth-generation grandparent of the head") download the state tree from the state root of p100(h), using the hashlookup opcode (note that after the first one or two rounds, this can be parallelized among as many peers as desired). verify that all parts of the tree match up. proceed normally from there. for light clients, the state root is even more advantageous: they can immediately determine the exact balance and status of any account by simply asking the network for a particular branch of the tree, without needing to follow bitcoin's multi-step 1-of-n "ask for all transaction outputs, then ask for all transactions spending those outputs, and take the remainder" light-client model. however, this state tree mechanism has an important disadvantage if implemented naively: the intermediate nodes in the tree greatly increase the amount of disk space required to store all the data. to see why, consider this diagram here: the change in the tree during each individual block is fairly small, and the magic of the tree as a data structure is that most of the data can simply be referenced twice without being copied. however, even still, for every change to the state that is made, a logarithmically large number of nodes (ie. ~5 at 1000 nodes, ~10 at 1000000 nodes, ~15 at 1000000000 nodes) need to be stored twice, one version for the old tree and one version for the new trie. eventually, as a node processes every block, we can thus expect the total disk space utilization to be, in computer science terms, roughly o(n*log(n)), where n is the transaction load. in practical terms, the ethereum blockchain is only 1.3 gigabytes, but the size of the database including all these extra nodes is 10-40 gigabytes. so, what can we do? one backward-looking fix is to simply go ahead and implement headers-first syncing, essentially resetting new users' hard disk consumption to zero, and allowing users to keep their hard disk consumption low by re-syncing every one or two months, but that is a somewhat ugly solution. the alternative approach is to implement state tree pruning: essentially, use reference counting to track when nodes in the tree (here using "node" in the computer-science term meaning "piece of data that is somewhere in a graph or tree structure", not "computer on the network") drop out of the tree, and at that point put them on "death row": unless the node somehow becomes used again within the next x blocks (eg. x = 5000), after that number of blocks pass the node should be permanently deleted from the database. essentially, we store the tree nodes that are part of the current state, and we even store recent history, but we do not store history older than 5000 blocks. x should be set as low as possible to conserve space, but setting x too low compromises robustness: once this technique is implemented, a node cannot revert back more than x blocks without essentially completely restarting synchronization. now, let's see how this approach can be implemented fully, taking into account all of the corner cases: when processing a block with number n, keep track of all nodes (in the state, tree and receipt trees) whose reference count drops to zero. place the hashes of these nodes into a "death row" database in some kind of data structure so that the list can later be recalled by block number (specifically, block number n + x), and mark the node database entry itself as being deletion-worthy at block n + x. if a node that is on death row gets re-instated (a practical example of this is account a acquiring some particular balance/nonce/code/storage combination f, then switching to a different value g, and then account b acquiring state f while the node for f is on death row), then increase its reference count back to one. if that node is deleted again at some future block m (with m > n), then put it back on the future block's death row to be deleted at block m + x. when you get to processing block n + x, recall the list of hashes that you logged back during block n. check the node associated with each hash; if the node is still marked for deletion during that specific block (ie. not reinstated, and importantly not reinstated and then re-marked for deletion later), delete it. delete the list of hashes in the death row database as well. sometimes, the new head of a chain will not be on top of the previous head and you will need to revert a block. for these cases, you will need to keep in the database a journal of all changes to reference counts (that's "journal" as in journaling file systems; essentially an ordered list of the changes made); when reverting a block, delete the death row list generated when producing that block, and undo the changes made according to the journal (and delete the journal when you're done). when processing a block, delete the journal at block n x; you are not capable of reverting more than x blocks anyway, so the journal is superfluous (and, if kept, would in fact defeat the whole point of pruning). once this is done, the database should only be storing state nodes associated with the last x blocks, so you will still have all the information you need from those blocks but nothing more. on top of this, there are further optimizations. particularly, after x blocks, transaction and receipt trees should be deleted entirely, and even blocks may arguably be deleted as well although there is an important argument for keeping some subset of "archive nodes" that store absolutely everything so as to help the rest of the network acquire the data that it needs. now, how much savings can this give us? as it turns out, quite a lot! particularly, if we were to take the ultimate daredevil route and go x = 0 (ie. lose absolutely all ability to handle even single-block forks, storing no history whatsoever), then the size of the database would essentially be the size of the state: a value which, even now (this data was grabbed at block 670000) stands at roughly 40 megabytes the majority of which is made up of accounts like this one with storage slots filled to deliberately spam the network. at x = 100000, we would get essentially the current size of 10-40 gigabytes, as most of the growth happened in the last hundred thousand blocks, and the extra space required for storing journals and death row lists would make up the rest of the difference. at every value in between, we can expect the disk space growth to be linear (ie. x = 10000 would take us about ninety percent of the way there to near-zero). note that we may want to pursue a hybrid strategy: keeping every block but not every state tree node; in this case, we would need to add roughly 1.4 gigabytes to store the block data. it's important to note that the cause of the blockchain size is not fast block times; currently, the block headers of the last three months make up roughly 300 megabytes, and the rest is transactions of the last one month, so at high levels of usage we can expect to continue to see transactions dominate. that said, light clients will also need to prune block headers if they are to survive in low-memory circumstances. the strategy described above has been implemented in a very early alpha form in pyeth; it will be implemented properly in all clients in due time after frontier launches, as such storage bloat is only a medium-term and not a short-term scalability concern. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum scalability and decentralization updates | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum scalability and decentralization updates posted by vitalik buterin on february 18, 2014 research & development scalability is now at the forefront of the technical discussion in the cryptocurrency scene. the bitcoin blockchain is currently over 12 gb in size, requiring a period of several days for a new bitcoind node to fully synchronize, the utxo set that must be stored in ram is approaching 500 mb, and continued software improvements in the source code are simply not enough to alleviate the trend. with every passing year, it becomes more and more difficult for an ordinary user to locally run a fully functional bitcoin node on their own desktop, and even as the price, merchant acceptance and popularity of bitcoin has skyrocketed the number of full nodes in the network has essentially stayed the same since 2011. the 1 mb block size limit currently puts a theoretical cap on this growth, but at a high cost: the bitcoin network cannot process more than 7 transactions per second. if the popularity of bitcoin jumps up tenfold yet again, then the limit will force the transaction fee up to nearly a dollar, making bitcoin less useful than paypal. if there is one problem that an effective implementation of cryptocurrency 2.0 needs to solve, it is this. the reason why we in the cryptocurrency spaceare having these problems, and are making so little headway toward coming up with a solution, is that there one fundamental issue with all cryptocurrency designs that needs to be addressed. out of all of the various proof of work, proof of stake and reputational consensus-based blockchain designs that have been proposed, not a single one has managed to overcome the same core problem: that every single full node must process every single transaction. having nodes that can process every transaction, even up to a level of thousands of transactions per second, is possible; centralized systems like paypal, mastercard and banking servers do it just fine. however, the problem is that it takes a large quantity of resources to set up such a server, and so there is no incentive for anyone except a few large businesses to do it. once that happens, then those few nodes are potentially vulnerable to profit motive and regulatory pressure, and may start making theoretically unauthorized changes to the state, like giving themselves free money, and all other users, which are dependent on those centralized nodes for security, would have no way of proving that the block is invalid since they do not have the resources to process the entire block. in ethereum, as of this point, we have no fundamental improvements over the principle that every full node must process every transaction. there have been ingenious ideas proposed by various bitcoin developers involving multiple merge-mined chains with a protocol for moving funds from one chain to another, and these will be a large part of our cryptocurrency research effort, but at this point research into how to implement this optimally is not yet mature. however, with the introduction of block protocol 2.0 (bp2), we have a protocol that, while not getting past the fundamental blockchain scalability flaw, does get us partway there: as long as at least one honest full node exists (and, for anti-spam reasons, has at least 0.01% mining power or ether ownership), “light clients” that only download a small amount of data from the blockchain can retain the same level of security as full nodes. what is a light client? the basic idea behind a light client is that, thanks to a data structure present in bitcoin (and, in a modified form, ethereum) called a merkle tree, it is possible to construct a proof that a certain transaction is in a block, such that the proof is much smaller than the block itself. right now, a bitcoin block is about 150 kb in size; a merkle proof of a transaction is about half a kilobyte. if bitcoin blocks become 2 gb in size, the proofs might expand to a whole kilobyte. to construct a proof, one simply needs to follow the “branch” of the tree all the way up from the transaction to the root, and provide the nodes on the side every step of the way. using this mechanism, light clients can be assured that transactions sent to them (or from them) actually made it into a block. this makes it substantially harder for malicious miners to trick light clients. if, in a hypothetical world where running a full node was completely impractical for ordinary users, a user wanted to claim that they sent 10 btc to a merchant with not enough resources to download the entire block, the merchant would not be helpless; they would ask for a proof that a transaction sending 10 btc to them is actually in the block. if the attacker is a miner, they can potentially be more sophisticated and actually put such a transaction into a block, but have it spend funds (ie. utxo) that do not actually exist. however, even here there is a defense: the light client can ask for a second merkle tree proof showing that the funds that the 10 btc transaction is spending also exist, and so on down to some safe block depth. from the point of view of a miner using a light client, this morphs into a challenge-response protocol: full nodes verifying transactions, upon detecting that a transaction spent an output that does not exist, can publish a “challenge” to the network, and other nodes (likely the miner of that block) would need to publish a “response” consisting of a merkle tree proof showing that the outputs in question do actually exist in some previous block. however, there is one weakness in this protocol in bitcoin: transaction fees. a malicious miner can publish a block giving themselves a 1000 btc reward, and other miners running light clients would have no way of knowing that this block is invalid without adding up all of the fees from all of the transactions themselves; for all they know, someone else could have been crazy enough to actually add 975 btc worth of fees. bp2 with the previous block protocol 1.0, ethereum was even worse; there was no way for a light client to even verify that the state tree of a block was a valid consequence of the parent state and the transaction list. in fact, the only way to get any assurances at all was for a node to run through every transaction and sequentially apply them to the parent state themselves. bp2, however, adds some stronger assurances. with bp2, every block now has three trees: a state tree, a transaction tree, and a stack trace tree providing the intermediate root of the state tree and the transaction tree after each step. this allows for a challenge-response protocol that, in simplified form, works as follows: miner m publishes block b. perhaps the miner is malicious, in which case the block updates the state incorrectly at some point. light node l receives block b, and does basic proof of work and structural validity checks on the header. if these checks pass, then l starts off treating the block as legitimate, though unconfirmed. full node f receives block b, and starts doing a full verification process, applying each transaction to the parent state, and making sure that each intermediate state matches the intermediate state provided by the miner. suppose that f finds an inconsistency at point k. then, f broadcasts a “challenge” to the network consisting of the hash of b and the value k. l receives the challenge, and temporarily flags b as untrustworthy. if f’s claim is false, and the block is valid at that point, then m can produce a proof of localized consistency by showing a merkle tree proof of point k in the stack trace, point k+1 in the stack trace, and the subset of merkle tree nodes in the state and transaction tree that were modified during the process of updating from k to k+1. l can then verify the proof by taking m’s word on the validity of the block up to point k, manually running the update from k to k+1 (this consists of processing a single transaction), and making sure the root hashes match what m provided at the end. l would, of course, also check that the merkle tree proof for the values at state k and k+1 is valid. if f’s claim is true, then m would not be able to come up with a response, and after some period of time l would discard b outright. note that currently the model is for transaction fees to be burned, not distributed to miners, so the weakness in bitcoin’s light client protocol does not apply. however, even if we decided to change this, the protocol can easily be adapted to handle it; the stack trace would simply also keep a running counter of transaction fees alongside the state and transaction list. as an anti-spam measure, in order for f’s challenge to be valid, f needs to have either mined one of the last 10000 blocks or have held 0.01% of the total supply of ether for at least some period of time. if a full node sends a false challenge, meaning that a miner successfully responds to it, light nodes can blacklist the node’s public key. altogether, what this means is that, unlike bitcoin, ethereum will likely still be fully secure, including against fraudulent issuance attacks, even if only a small number of full nodes exist; as long as at least one full node is honest, verifying blocks and publishing challenges where appropriate, light clients can rely on it to point out which blocks are flawed. note that there is one weakness in this protocol: you now need to know all transactions ahead of time before processing a block, and adding new transactions requires substantial effort to recalculate intermediate stack trace values, so the process of producing a block will be more inefficient. however, it is likely possible to patch the protocol to get around this, and if it is possible then bp2.1 will have such a fix. blockchain-based mining we have not finalized the details of this, but ethereum will likely use something similar to the following for its mining algorithm: let h[i] = sha3(sha3(block header without nonce) ++ nonce ++ i) for i in [0 ...16] let n be the number of transactions in the block. let t[i] be the (h[i] mod n)th transaction in the block. let s be the parent block state. apply t[0] ... t[15] to s, and let the resulting state be s'. let x = sha3(s'.root) the block is valid if x * difficulty <= 2^256 this has the following properties: this is extremely memory-hard, even more so than dagger, since mining effectively requires access to the entire blockchain. however it is parallelizable with shared disk space, so it will likely be gpu-dominated, not cpu-dominated as dagger originally hoped to be. it is memory-easy to verify, since a proof of validity consists of only the relatively small subset of patricia nodes that are used while processing t[0] ... t[15] all miners essentially have to be full nodes; asking the network for block data for every nonce is prohibitively slow. thus there will be a larger number of full nodes in ethereum than in bitcoin. as a result of (3), one of the major motivations to use centralized mining pools, the fact that they allow miners to operate without downloading the entire blockchain, is nullified. the other main reason to use mining pools, the fact that they even out the payout rate, can be assomplished just as easily with the decentralized p2pool (which we will likely end up supporting with development resources) asics for this mining algorithm are simultaneously asics for transaction processing, so ethereum asics will help solve the scalability problem. from here, there is only really one optimization that can be made: figuring out some way to get past the obstacle that every full node must process every transaction. this is a hard problem; a truly scalable and effective solution will take a while to develop. however, this is a strong start, and may even end up as one of the key ingredients to a final solution. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements announcement on planned withdrawal from exodus | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search announcement on planned withdrawal from exodus posted by vitalik buterin on august 8, 2014 research & development now that the first two weeks of the ether sale are over, and over 50 million eth has been sold, we intend to soon make a transaction to begin using the funds to repay loans and kickstart the process of setting up our development hubs and expanding our staff. we have made a commitment to be highly transparent about how we spend the funds, and that is a commitment that we intend to live up to; to that end, we have already released an intended use of revenue chart and a roadmap to show how we intend to spend the btc. more recently, the community has followed up with a wonderful infographic on cointelegraph using the information that we have posted. now, we intend to release some additional information about the nature of our first withdrawal transaction. the intent is to withdraw 4150 btc from our exodus address within the next 48 hours. we reserve the right to withdraw up to 850 btc more if needed before the end of the 42 day duration of the sale, but at this point it is likely that the remainder of the btc in the address will remain unused until the sale ends. of this amount, 2650 btc will be distributed to pay for loans for prior expenses. individuals who have contributed loans to the project will receive repayment in btc directly; "we" will not be selling any portion of this 2650 btc on exchanges ourselves, although individuals may choose to independently convert the btc that they receive into fiat after the fact. individuals also have the choice of taking the repayment in ether; in those cases, we will simply not send the btc, and once all repayments have been processed we will publish all of the additional eth that has been sold in this way (note that this is equivalent to sending individuals their btc and letting the recipients send it right back into the exodus). the remaining 1500 btc will be sent to a wallet controlled by đξv, our development arm, and will be used to establish our sites in berlin and amsterdam and begin hiring developers; some of this amount may be converted into eur, gbp or chf (eg. to pay for rent), and the remainder will be held as btc. the following spreadsheet provides a rough categorization of how the backpay and forward-pay expenses are to be ultimately distributed. https://docs.google.com/a/ethereum.org/spreadsheets/d/1yqymlknf9tibarjyrkhef-ivnmga6ffvhjnqh_no_ao/edit#gid=0 the largest category is pay for individuals, covering core developers, web developers and art, communications, branding and business development, and among the expenses the largest is legal at 296,000followedbyrentat296,000 followed by rent at 296,000followedbyrentat111,000 (including a $16,500 security deposit which is theoretically refundable and pre-payment up to feb 2015) and the other categories you can see for yourself by looking at the chart. going forward, the primary change is that expenditures are now going to be much more focused on paying for development. our intent is to have our development centers in berlin and amsterdam, with a smaller presence in toronto and london to cover communications, marketing and branding; the extent of our presence in san francisco / silicon valley and possibly other locations is still to be determined and will be done based on cost-benefit analysis. additionally, note that the distribution of the endowment is quasi-public; although names of all individuals are not published (though everyone is free to disclose their own portion voluntarily, and the owners of the largest pieces can be partially inferred from public information), the percentages are available for view at https://docs.google.com/spreadsheets/d/1gs9pzsdmx9lk0xgskedr_aoi02riq3mpryventvum68/edit#gid=0. in the future, we intend to continue to uphold and step up our commitment to transparency, releasing details on how funds are being spent and on the progress of the project; if you are interested, feel free to follow our blog and the public blockchain. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements decentralized protocol monetization and forks | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search decentralized protocol monetization and forks posted by vitalik buterin on april 30, 2014 research & development the idea of releasing a new currency as a mechanism for funding protocol development is perhaps one of the most interesting economic innovations to come out of the cryptocurrency space. in the past twenty years, we have seen a growing centralization in the protocols that underlie the internet, with the rise of proprietary chat systems and social networks like facebook, and a large part of the reason for this trend has been the need for monetization; if facebook was cryptographically secure and decentralized, the developers would have no way to make money by data mining their users’ activities and taking a 30% cut of their internal currency, and so decentralized alternatives to facebook have largely fizzled due to lack of institutional support and funding. with decentralized protocols, however, we have discovered a new mechanism for monetizing them: create internal assets, and sell them to pay for the development of the protocol. in general, so far we know of two classes of “internal assets” that can be sold in this way; first, there is the idea of creating an internal token system, a crypto-fuel with a floating price that has some value in the network, and second, one can introduce name registrations; for example, a decentralized twitter might fund itself by building in its own decentralized username registration mechanism similar to namecoin and selling off the 1-4 letter names. this new monetization model is powerful, and in the first of the two above-described implementations already has a number of proven successes, but it is also incredibly non-intrusive – it requires no licensing schemes, proprietary software, crippleware or privacy infringement, and in fact no one actually has to explicitly “pay” for anything at all (if you buy tokens you are just swapping into a different asset, which can easily hold its value against other assets). however, in this model there is one concern that many people have raised, and that is the question of forks. in short, if one releases a new decentralized protocol that is based on a token system, why won’t someone else release a fork with either their own token system, or a token system that is somehow tied to an asset with an existing userbase, and if one releases a decentralized twitter with a built-in name registration system why won’t someone release a fork that points to their own name registration system, or even the original namecoin? in traditional business, there are two solutions to the problem. one is to give up the idea of making everything open-source, and keep at least the latest version of the client proprietary. the other is to release the protocol for free, and then sell services. of course, both approaches have their own very well-understood flaws. in the context of a decentralized blockchain application, most of the benefits of decentralization are lost when the code becomes proprietary – with a proprietary mining algorithm, for example, there is no way to prove that it does not have a backdoor for its developers, and is therefore equivalent to the developers simply running a centralized server and asking the community to trust them. the second approach, selling services, is also flawed; first, the revenue is in most cases vastly insufficient, and second, it incentivizes the organization to produce only a minimal decentralized protocol in order to then sell centralized services on top, rather than building up an entire decentralized ecosystem. many decentralized projects are pursuing neither of these strategies; for example, ethereum itself is 100% open source, and have been since even before the day that it publicly launched. many protocol organizations, including our own, are interested in transforming themselves into “decentralized autonomous organizations”, which necessarily implies a very high degree of transparency. given this, what is a decentralized protocol’s “moat” against forks? what stops another group from taking all of our code and research ready-made and creating their own version of the blockchain, perhaps with one or two superior features (or simply having a large endowment and dumping it all into superior marketing), and taking us over? the question is a difficult one, but it has a number of interesting answers, both in terms of ethereum specifically and decentralized protocols as a whole. on flimsy moats and dictators in order to answer the question, it is important to first understand that, in the space of tech companies and especially social networking startups, a large number of them are literally backed by almost nothing but social consensus. theoretically, it is entirely possible for all of the employees at snapchat, tinder, twitter or any other such startup to all suddenly agree to quit and start their own business, completely rebuild all of the software from scratch within months, and then immediately proceed to build a superior product. the only reason why such companies have any valuation at all is a set of two coordination problems: the problem of getting all employees to quit at the same time, and the problem of getting all of the customers to simultaneously move over onto the new network. in the context of a service like dropbox, the latter issue does not exist; because dropbox is just as useful to each individual if one other person is using it or a million, there is no reason why people can’t move over a few at a time. in the context of a social network, which is useless unless everyone else is already on it, the problem is fundamental. in the abstract, this may seem like a flimsy justification for why tech companies are valuable; when thinking about something that represents billions of dollars of value, one naturally expects that value to be backed up by something tangible like physical resources or government force, not just some ethereal instantiation of the fact that it’s hard for large groups of people to suddenly move from one social configuration to another. in reality, however, even physical resources and government force are backed by nothing but a social coordination problem – if 70% of the victims of a dictatorship were to simultaneously rise up against their dictator, the government would get toppled pretty quickly, and yet most dictators even running rather brutally oppressive regimes are quite comfortable sitting in their lofty thrones knowing that such a thing will almost certainly not happen. given this background in theory, what exactly are the social coordination problems backing up a decentralized blockchain? what exactly is the “moat” that is backing up the value of the “official” ethereum blockchain or mastercoin state transition system, and ether as a mechanism of storing value and paying for transaction fees, as opposed to alternate clones like “aethereum“? specifically, what are the necessary factors that make the original version of a given decentralized protocol superior, when all of its underlying features can easily be cloned, and even improved upon as soon as a group discovers even one flaw in the original (in the case of bitcoin, for example, one can trivially improve the bitcoin protocol by removing the requirement for multisig spending transactions to have an extraneous zero in the spending script code, an anti-feature which was introduced accidentally)? as it turns out, there is quite a lot. teams first of all, every project has a core development team. in fact, this aspect is actually stronger in the case of a decentralized token system than a traditional tech company. while in a traditional tech company, there might be only a very small number of people with shares in the company and who are thus incentivized to stick with it and see it succeed, in the case of a decentralized token system there are dozens or even hundreds of people holding tokens associated with the project; in fact, many people actually choose to be paid predominantly in tokens. in the case of ethereum, for example, the size of the list of people who will be receiving ether as compensation for work done currently stands at sixty-eight, and will increase even further as time goes on. and all of these tokens are, of course, untradeable until the protocol actually launches, so all of the token holders are strongly incentivized to do their best to ensure that the system does as well as possible. thus, the team, the set of people who know the most about how the protocol works from the experience of having actually developed it, is a decentralized project’s core asset that competitive spinoffs cannot so easily “fork” and replicate, and it is the team that will be responsible for much of the rest of the project’s “moat”. network effects of exposure the simplest reason why people will use the original blockchain and not a fork is simple: it’s the default. people hear about bitcoin first, so they go to bitcoin.org and download the bitcoin client, and use bitcoin to buy and sell goods and services, notbitcoin scrypt. for the same reason, people use the official version of most open-source projects and not any of the thousands of forks, buy music, books and movies instead of trying to download them via torrents, and use popular bitcoin wallets instead of less popular ones. any fork of a given protocol necessarily comes after the original, and is therefore much less likely to gain media attention. moral pressure another important reason why the original version of a protocol is more likely to gain media attention than a fork is plain old public morality: people believe that the developers of a project deserve to get compensated, and so a fork which is developed with the primary purpose of depriving the developers of compensation is likely to be viewed negatively, or at least less favorably, by many people. this moral effect can be a very powerful one, and contributes heavily to the original protocol’s greater exposure; the best empirical evidence for this is likely the success of services like netflix over filesharing-based alternatives. at the same time, however, if the original developers of a protocol start taking development in an undesirable direction (eg. introducing backdoors, introducing excessively intrusive monetization vehicles, or even just being too plain slow), then the moral effect can rapidly turn on its head and even support the first credible effort to try to wrest away a project from its creators; following the prior example, the pertinent example here is the media success of the pirate bay and popcorn time. thus, moral pressure can work both for and against a decentralized protocol, and it is the protocol developers’ responsibility to ensure that the community opinion of their project remains positive, and serves as an important check-and-balance to make sure that the core team behind a project continues to move the project forward at a solid pace and in an agreeable direction. network effects of currency unit liquidity one argument that is often raised against forks of bitcoin is the idea of liquidity, or specifically market depth: smaller currencies are inherently weaker than larger currencies because there are fewer people buying and selling them, and so you will move the price much more if you try to sell a large amount. however, this argument is only important up to a certain point; once a currency reaches a sufficient size, it has enough market depth to cover all ordinary usage, and so additional depth provides little value. hence, this network effect provides a moderately strong edge against forks with a new token system, which will have very low market depth to start off, although at the cost of a slight disadvantage against forks that tie in existing large currencies via two-way-pegging mechanisms. ecosystemic network effects an important feature of decentralized protocols, and social protocols in general, is that they also build ecosystems. on a social network, for example, there is a one-dimensional network effect: a social network is more useful if more people use it. with a currency, that effect becomes two-dimensional: a currency attracts more users if there are more merchants, and more merchants if there are more users. once development effort, security and liquidity come into play, this increases to three to six dimensions. all of these interdependencies make it hard for a new version of a social network to bore its way into mainstream acceptance, as initially it starts off with nothing. in the case of ethereum, the tightly integrated nature of the currency system actually makes the network effect in some respects highly multi-dimensional. the relevant property of the ethereum architecture is the first-class-citizen property of contracts: contracts can interact with, send and receive messages from and hold accounts with other contracts much like external accounts can. this allows you to cleverly pull together long chains of contracts and applications, using contracts of different types at each step of the interaction process. for example, i might hold some shares of a decentralized autonomous organization (contract a), where the shares are held on a decentralized market (contract b) in a multisignature account (contract c) for added security. the co-signer of said multisig account is paranoid about quantum computing, so he uses custom cryptography (contract d) based on verifying lamport signatures for authentication. the organization would then store some of its funds in a usd-pegged asset using a financial derivatives market (contract f) using a combination of centralized and decentralized data feeds (contracts g, h, i), and internally uses a name registration system (contract j) to store all of the functions that it calls. a single transaction may end up calling all of these contracts multiple times. liquid markets for on-blockchain assets, liquid markets for message publication, and a robust ecosystem of daos, decentralized exchanges, financial markets and data feeds all support each other and make the ethereum blockchain stronger. the ethereum blockchain is not just a blockchain; it’s really one large decentralized computer where all of the components are tightly linked together, and each component provides additional tools for other components to play with. bugs and attacks this is a small point, but an important one. there is always a risk that either the protocol or the client implementation will be flawed in some way. as hard as the bitcoin developers have tried, the bitcoind source code has had problems crop up over the years, and twice in bitcoin’s history (specifically, the integer overflow exploit in 2010 and the fork in 2013) such problems have even led to a consensus failure that required manual resolution. in theory, developers of every protocol try as hard as they can to ensure that bugs never happen in the first place. in practice, of course, there is always a chance that something will slip by, the price will start crashing ten or twenty percent within an hour, and it will be up to the developers, the miners and the large businesses to quickly push out and coordinate a fix. sometimes, such errors may not even be the protocol’s fault; a massive megacorporate or government-sponsored 51% attack or a globally coordinated distributed denial of service on the entire network are also possibilities, and might need special measures to be dealt with. thus, as decentralized as peer to peer protocols aspire to be, ultimately they do benefit considerably from some degree of institutional support in times of crisis – support that the original developers who understand the protocol and software best are the best-equipped to provide. protocol upgrades ethereum 1.0 is far from perfect, and between our discussions on the development roadmap and the hard problems of cryptocurrency we have been very open about admitting this. there are plenty of ways that blockchain technology could be improved, ranging from research on price-stabilized currencies to better fee structures, alternative consensus models and, as a holy grail, multi-blockchain architectures or scip. however, the intricacies of actually coming up with the math and then implementing these mechanisms, are in many cases even figuring out whether or not they are even possible, are sufficiently complex that we have decided there is a large list of features we are simply not going to do for ethereum 1.0. to that end, we have established the long-term roadmap that we will release ethereum 1.0 in q4 2014 at the latest, and at the same time we have already started to set up efforts to research the kinds of improvements that we can theoretically add, specifically in terms of scalability, with a plan to crystallize them into ethereum 2.0 at some point around 2016. ethereum 2.0 will use “ether 2.0″ as its currency, where the main initial mechanism for obtaining a unit of ether 2.0 is simply to provably destroy a unit of ether 1.0. thus, the currency inside of a protocol is backed not just by the utility and network effects of the current implementation of that protocol, but also the promise of better future versions of the protocol to come. of course, cryptocurrency protocols are hard to change, and in practice bitcoin has proven very difficult to change in the short term, but more large-scale re-architectures are actually somewhat easier to implement than small changes when one looks at the ratio of effort to effect. we have already seen the master protocol make several upgrades, and we will likely see ethereum 2.0, 3.0 and perhaps even further over the next few years and decades. what’s the point? finally, the most important argument of all is, what’s the point of a fork? in the case of bitcoin, there are many reasons to fork the code – you might want to add support for more transaction types, change the currency supply, replace the currency with a centralized alternative backed by the us dollar, or change the type of cryptography used. if a protocol is correctly generalized, however, there simply is no way to improve that can’t be replicated inside the protocol itself. for example, if you are using ripple then you can use ripple equally easily to store xrp, cryptocurrencies, fiat currencies, local community currencies or little bobby’s magic token points. hence, concerns about optimal monetary policy, politicization or depoliticization of money or many of the other debates surrounding bitcoin have no bearing on the success of the ripple protocol itself. in the case of ethereum, the protocol has a generic programming language, making the system even more malleable: if someone comes up with a blockchain-based system that is better than ethereum in some fashion (with the exception of secure near-instant block times), then someone else can fork it right back inside of ethereum itself by simply implementing it as a contract. this fork would immediately benefit from ethereum’s ecosystemic network effects, allowing users to benefit from both the superior feature and the ability to interface seamlessly and directly with an existing ecosystem of liquid markets, data feeds and daos. using this power of the contract mechanism, ethereum will be able to contain side-chains of bitcoin, litecoin and dogecoin (yes, even scrypt-based coins can be turned into side-chains via computational stacktraces and an economically incentivized challenge-response protocol), name registrations, post-quantum cryptography and an unlimited number of other features. thus, on the whole decentralized protocols lie in an interesting place in the modern economy. on the one hand, much like bitcoin itself, they are in a very clear way “backed by nothing”. on the other hand, they actually have quite a powerful backing underneath, and one that is difficult to unseat; in practice, we have seen very few examples of any open source software fork unseating the original, both in the cryptocurrency space and outside of it. nothing has unseated bitcoin, nothing has unseated litecoin and nothing has unseated dogecoin. the only forks that do gain serious community acceptance are the ones that add a large body of new features, and these forks always succeed in carving out a niche of their own. fortunately, we still have many decades to go in seeing exactly how the decentralized protocol ecosystem is going to play out. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements visions, part 2: the problem of trust | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search visions, part 2: the problem of trust posted by vitalik buterin on april 27, 2015 research & development special thanks to: robert sams, gavin wood, mark karpeles and countless cryptocurrency critics on online forums for helping to develop the thoughts behind this article if you were to ask the average cryptocurrency or blockchain enthusiast what the key single fundamental advantage of the technology is, there is a high chance that they will give you one particular predictable answer: it does not require trust. unlike traditional (financial or other) systems, where you need to trust a particular entity to maintain the database of who holds what quantity of funds, who owns a particular internet-of-things-enabled device, or what the status is of a particular financial contract, blockchains allow you to create systems where you can keep track of the answers to those questions without any need to trust anyone at all (at least in theory). rather than being subject to the whims of any one arbitrary party, someone using a blockchain technology can take comfort in the knowledge that the status of their identity, funds or device ownership is safely and securely maintained in an ultra-secure, trustless distributed ledger backed by math™. contrasting this, however, there is the standard critique that one might hear on forums like buttcoin: what exactly is this "trust problem" that people are so worried about? ironically enough, unlike in "crypto land", where exchanges seem to routinely disappear with millions of dollars in customer funds, sometimes after apparently secretly being insolvent for years, businesses in the real world don't seem to have any of these problems. sure, credit card fraud exists, and is a major source of worry at least among americans, but the total global loss is a mere $190 billion less than 0.4% of global gdp, compared to the mtgox loss that seems to have cost potentially more than the value of all bitcoin transactions in that year. at least in the developed world, if you put your money in a bank, it's safe; even if the bank goes awry, your funds are in most cases protected up to over $100,000 by your national equivalent of the fdic even in the case of the cyprus depositor haircut, everything up to the deposit insurance limit was kept intact. from such a perspective, one can easily see how the traditional "centralized system" is serving people just fine. so what's the big deal? trust first, it is important to point out that distrust is not nearly the only reason to use blockchains; i mentioned some much more mundane use cases in the previous part of this series, and once you start thinking of the blockchain simply as a database that anyone can read any part of but where each individual user can only write to their own little portion, and where you can also run programs on the data with guaranteed execution, then it becomes quite plausible even for a completely non-ideological mind to see how the blockchain might eventually take its place as a rather mundane and boring technology among the likes of mongodb, angularjs and continuation-based web servers by no means even close to as revolutionary as the internet itself, but still quite powerful. however, many people are interested in blockchains specifically because of their property of "trustlessness", and so this property is worth discussing. to start off, let us first try to demystify this rather complicated and awe-inspiring concept of "trust" and, at the same time, trustlessness as its antonym. what exactly is trust? dictionaries in this case tend not to give particularly good definitions; for example, if we check wiktionary, we get: confidence in or reliance on some person or quality: he needs to regain her trust if he is ever going to win her back. dependence upon something in the future; hope. confidence in the future payment for goods or services supplied; credit: i was out of cash, but the landlady let me have it on trust. there is also the legal definition: a relationship created at the direction of an individual, in which one or more persons hold the individual's property subject to certain duties to use and protect it for the benefit of others. neither is quite precise or complete enough for our purposes, but they both get us pretty close. if we want a more formal and abstract definition, we can provide one as follows: trust is a model of a particular person or group's expected behavior, and the adjustment of one's own behavior in accordance with that model. trust is a belief that a particular person or group will be affected by a particular set of goals and incentives at a particular time, and the willingness to take actions that rely on that model being correct. just from the more standard dictionary definition, one may fall into the trap of thinking that trust is somehow inherently illogical or irrational, and that one should strive hard to trust as little as possible. in reality, however, can see that such thinking is completely fallacious. everyone has beliefs about everything; in fact, there are a set of theorems which basically state that if you are a perfectly rational agent, you pretty much have to have a probability in your head for every possible claim and update those probabilities according to certain rules. but then if you have a belief, it is irrational not to act on it. if, in your own internal model of the behavior of the individuals in your local geographic area, there is a greater than 0.01% chance that if you leave your door unlocked, someone will steal $10000 worth of goods from your house, and you value the inconvenience of carrying your key around at $1, then you should lock your door and bring the key along when you go to work. but if there is a less than 0.01% chance that someone will come in and steal that much, it is irrational to lock the door. "trustlessness" in its absolute form does not exist. given any system that is maintained by humans, there exists a hypothetical combination of motivations and incentives that would lead those humans to successfully collude to screw you over, and so if you trust the system to work you are necessarily trusting the total set of humans not to have that particular combination of motivations and incentives. but that does not mean that trustlessness is not a useful direction to strive in. when a system is claiming to be "trustless", what it is actually trying to do is expand the possible set of motivations that individuals are allowed to have while still maintaining a particular low probability of failure. when a system is claiming to be "trustful", it is trying to reduce the probability of failure given a particular set of motivations. thus, we can see the "trustlessness" and "trustfulness", at least as directions, are actually the exact same thing: note that in practice the two may be different connotatively: "trustless" systems tend to try harder to improve system trustability given a model where we know little about individuals' motivations, and "trustful" systems tend to try harder to improve system trustability given a model where we know a lot about individuals' motivations, and we know that those motivations are with higher probability honest. both directions are likely worthwhile. another important point to note is that trust is not binary, and it is not even scalar. rather, it is of key importance what it is that you are trusting people to do or not to do. one particular counterintuitive point is that it is quite possible, and often happens, that we trust someone not to do x, but we don't trust them not to do y, even though that person doing x is worse for you than them doing y. you trust thousands of people every day not to suddenly whip a knife out of their pockets as you pass by and stab you to death, but you do not trust complete strangers to hold on to $500 worth of cash. of course, the reason why is clear: no one has an incentive to jump out at you with a knife, and there is a very strong disincentive, but if someone has your $500 they have a $500 incentive to run away with it, and they can quite easily never get caught (and if they do the penalties aren't that bad). sometimes, even if incentives in both cases are similar, such counterintuitive outcomes can come simply because you have nuanced knowledge of someone else's morality; as a general rule, you can trust that people are good at preventing themselves from doing things which are "obviously wrong", but morality does quite often fray around the edges where you can convince yourself to extend the envelope of the grey (see bruce schneier's concept of "moral pressures" in liars and outliers and dan ariely's the honest truth about dishonesty for more on this). this particular nuance of trust has direct relevance in finance: although, since the 2008 financial crisis, there has indeed been an upsurge in distrust in the financial system, the distrust that the public feels is not a feeling that there is a high risk that the bank will steal the people's assets blatantly and directly and overwrite everyone's bank balance to zero. that is certainly the worst possible thing that they could do to you (aside from the ceo jumping out at you when you enter the bank branch and stabbing you to death), but it is not a likely thing for them to do: it is highly illegal, obviously detectable and will lead to the parties involved going to jail for a long long time and, just as importantly, it is hard for the bank ceo to convince themselves or their daughter that they are still a morally upright person if they do something like that. rather, we are afraid that the banks will perform one of many more sneaky and mischievious tricks, like convincing us that a particular financial product has a certain exposure profile but hiding the black swan risks. even while we are always afraid that large corporations will do things to us that are moderately shady, we are at the same time quite sure that they won't do anything extremely outright evil at least most of the time. so where in today's world are we missing trust? what is our model of people's goals and incentives? who do we rely on but don't trust, who could we rely on but don't because we don't trust them, what exactly is it that we are fearing they would do, and how can decentralized blockchain technology help? finance there are several answers. first, in some cases, as it turns out, the centralized big boys still very much can't be trusted. in modern financial systems, particularly banks and trading systems, there exists a concept of "settlement" essentially, a process after a transaction or trade is made the final result of which is that the assets that you bought actually become yours from a legal property-ownership standpoint. after the trade and before settlement, all that you have is a promise that the counterparty will pay a legally binding promise, but even legal bonds count for nothing when the counterparty is insolvent. if a transaction nets you an expected profit of 0.01%, and you are trading with a company that you estimate has a chance of 1 in 10000 of going insolvent on any particular day, then a single day of settlement time makes all the difference. in international transactions, the same situation applies, except this time the parties actually don't trust each other's intentions, as they are in different jurisdictions and some operate in jurisdictions where the law is actually quite weak or even corrupt. back in the old days, legal ownership of securities would be defined by ownership of a piece of paper. now, the ledgers are electronic. but then, who maintains the electronic ledger? and do we trust them? in the financial industry more than anywhere else, the combination of a high ratio of capital-at-stake to expected-return and the high ability to profit from malfeasance means that trust risks are greater than perhaps almost any other legal white-market industry. hence, can decentralized trustworthy computing platforms and very specifically, politically decentralized trustworthy computing platforms, save the day? according to quite a few people, yes they can. however, in these cases, commentators such as tim swanson have pointed out a potential flaw with the "fully open" pow/pos approach: it is a little too open. in part, there may be regulatory issues with having a settlement system based on a completely anonymous set of consensus participants; more importantly, however, restricting the system can actually reduce the probability that the participants will collude and the system will break. who would you really trust more: a collection of 31 well-vetted banks that are clearly separate entities, located in different countries, not owned by the same investing conglomerates, and are legally accountable if they collude to screw you over, or a group of mining firms of unknown quantity and size with no real-world reputations, 90% of whose chips may be produced in taiwan or shenzhen? for mainstream securities settlement, the answer that most people in the world would give seems rather clear. but then, in ten years' time, if the set of miners or the set of anonymous stakeholders of some particular currency proves itself trustworthy, eventually banks may warm up to even the more "pure cryptoanarchic" model or they may not. interaction and common knowledge another important point is that even if each of us has some set of entities that we trust, not all of us have the same set of entities. ibm is perfectly fine trusting ibm, but ibm would likely not want its own critical infrastructure to be running on top of google's cloud. even more pertinently, neither ibm nor google may be interested in having their critical infrastructure running on top of tencent's cloud, and potentially increasing their exposure to the chinese government (and likewise, especially following the recent nsa scandals, there has been increasing interest in keeping one's data outside the us, although this must be mentioned with the caveat that much of the concern is about privacy, not protection against active interference, and blockchains are much more useful at providing the latter than the former). so, what if ibm and tencent want to build applications that interact with each other heavily? one option is to simply call each other's services via json-rpc, or some similar framework, but as a programming environment this is somewhat limited; every program must either live in ibm land, and take 500 milliseconds round-trip to send a request to tencent, or live in tencent land, and take 500 milliseconds to send a request to ibm. reliability also necessarily drops below 100%. one solution that may be useful in some cases is to simply have both pieces of code living on the same execution environment, even if each piece has a different administrator but then, the shared execution environment needs to be trusted by both parties. blockchains seem like a perfect solution, at least for some use cases. the largest benefits may come when there is a need for a very large number of users to interact; when it's just ibm and tencent, they can easily make some kind of tailored bilateral system, but when n companies are interacting with each other, you would need either n2 bilateral systems among every pair of companies, or you can more simply make a single shared system for everyone and that system might as well be called a blockchain. trust for the rest of us the second case for decentralization is more subtle. rather than concentrating on the lack of trust, here we emphasize the barrier to entry in becoming a locus of trust. sure, billion dollar companies can certainly become loci of trust just fine, and indeed it is the case that they generally work pretty well with a few important exceptions that we will discuss later on. however, their ability to do so comes at a high cost. although the fact that so many bitcoin businesses have managed to abscond with their customers' funds is sometimes perceived as a strike against the decentralized economy, it is in fact something quite different: it is a strike against a economy with low social capital. it shows that the high degree of trust that mainstream institutions have today is not something that simply arose because powerful people are especially nice and tech geeks are less nice; rather, it is the result of centuries of social capital built up over a process which would take many decades and many trillions of dollars of investment to replicate. quite often, the institutions only play nice because they are regulated by governments and the regulation itself is in turn not without large secondary costs. without that buildup of social capital, well, we simply have this: and lest you think that such incidents are a unique feature of "cryptoland", back in the real world we also have this: the key promise of decentralized technology, under this viewpoint, is not to create systems that are even more trustworthy than current large institutions; if one simply looks at basic statistics in the developed world, one can see that many such systems can quite reasonably be described as being "trustworthy enough", in that their annual rate of failure is sufficiently low that other factors dominate in the choice of which platform to use. rather, the key promise of decentralized technology is to provide a shortcut to let future application developers get there faster: traditionally, creating a service that holds critical customer data or large quantities of customer funds has involved a very high degree of trust, and therefore a very large degree of effort some of it involving complying with regulations, some convincing an established partner to lend you their brand name, some buying extremely expensive suits and renting fake "virtual office space" in the heart of downtown new york or tokyo, and some simply being an established company that has served customers well for decades. if you want to be entrusted with millions, well, better be prepared to spend millions. with blockchain technology, however, the exact opposite is potentially the case. a 5-of-8 multisig consisting of a set of random individuals around the world may well have a lower probability of failure than all but the largest of institutions and at a millionth of the cost. blockchain-based applications allow developers to prove that they are honest by setting up a system where they do not even have any more power than the users do. if a group of largely 20-to-25-year old college dropouts were to announce that they were opening a new prediction market, and asked people to deposit millions of dollars to them via bank deposit, they would likely be rightfully viewed with suspicion. with blockchain technology, on the other hand, they can release augur as a decentralized application, and they can assure the whole world that their ability to run away with everyone's funds is drastically reduced. particularly, imagine what would be the case if this particular group of people was based in india, afghanistan or, heck, nigeria. if they were not a decentralized application, they would likely not have been able to get anyone's trust at all. even in the developed world, the less effort you need to spend convincing users that you are trustworthy, the more you are free to work on developing your actual product. subtler subterfuge finally, of course, we can get back to the large corporations. it is indeed a truth, in our modern age, that large companies are increasingly distrusted they are increasingly distrusted by regulators, they are increasingly distrusted by the public, and they are increasingly distrusted by each other. but, at least in the developed world, it seems obvious that they are not going to go around zeroing out people's balances or causing their devices to fail in arbitrarily bad ways for the fun of it. so if we distrust these behemoths, what is it that we are afraid they will do? trust, as discussed above, isn't a boolean or a scalar, it's a model of someone else's projected behavior. so what are the likely failure modes in our model? the answer generally comes from the concept of base-layer services, as outlined in the previous part of this series. there are certain kinds of services which happen to have the property that they (1) end up having other services depending on them, (2) have high switching costs, and (3) have high network effects, and in these cases, if a private company operating a centralized service creates a monopoly they have substantial latitude over what they can do to protect their own interests and establish a permanent position for themselves at the center of society at the expense of everyone else. the latest incident that shows the danger came one week ago, when twitter cut video streaming service meerkat off of its social network api. meerkat's offense: allowing users to very easily import their social connections from twitter. when a service becomes a monopoly, it has the incentive to keep that monopoly. whether that entails disrupting the survival of companies that try to build on the platform in a way that competes with its offerings, or restricting access to users' data inside the system, or making it easy to come in but hard to move away, there are plenty of opportunities to slowly and subtly chip away at users' freedoms. and we increasingly do not trust companies not to do that. building on blockchain infrastructure, on the other hand, is a way for an application developer to commit not to be a jerk, forever. ... and laziness in some cases, there is also another concern: what if a particular service shuts down? the canonical example here is the various incarnations of "remindme" services, which you can ask to send you a particular message at some point in the future perhaps in a week, perhaps in a month, and perhaps in 25 years. in the 25-year case (and realistically even the 5-year case), however, all currently existing services of that kind are pretty much useless for a rather obvious reason: there is no guarantee that the company operating the service will continue to exist in 5 years, much less 25. not trusting people not to disappear is a no-brainer; for someone to disappear, they do not even have to be actively malicious they just have to be lazy. this is a serious problem on the internet, where 49% of documents cited in court cases are no longer accessible because the servers on which the pages were located are no longer online, and to that end projects like ipfs are trying to resolve the problem via a politically decentralized content storage network: instead of referring to a file by the name of the entity that controls it (which an address like "https://blog.ethereum.org/2015/04/13/visions-part-1-the-value-of-blockchain-technology/" basically does), we refer to the file by the hash of the file, and when a user asks for the file any node on the network can provide it in the project's own words, creating "the permanent web". blockchains are the permanent web for software daemons. this is particularly relevant in the internet of things space; in a recent ibm report, one of their major concerns with the default choice for internet of things infrastructure, a centralized "cloud", that they cite is as follows: while many companies are quick to enter the market for smart, connected devices, they have yet to discover that it is very hard to exit. while consumers replace smartphones and pcs every 18 to 36 months, the expectation is for door locks, led bulbs and other basic pieces of infrastructure to last for years, even decades, without needing replacement ... in the iot world, the cost of software updates and fixes in products long obsolete and discontinued will weigh on the balance sheets of corporations for decades, often even beyond manufacturer obsolescence. from the manufacturer's point of view, having to maintain servers to deal with remaining instances of obsolete products is an annoying expense and a chore. from the consumer's point of view, there is always the nagging fear: what if the manufacturer simply shrugs off this responsibility, and disappears without bothering to maintain continuity? having fully autonomous devices managing themselves using blockchain infrastructure seems like a decent way out. conclusion trust is a complicated thing. we all want, at least to some degree, to be able to live without it, and be confident that we will be able to achieve our goals without having to take the risk of someone else's bad behavior much like every farmer would love to have their crops blossom without having to worry about the weather and the sun. but economy requires cooperation, and cooperation requires dealing with people. however, impossibility of an ultimate end does not imply futility of the direction, and in any case it is always a worthwhile task to, whatever our model is, figure out how to reduce the probability that our systems will fail. decentralization of the kind described here is not prevalent in the physical world primarily because the duplication costs involved are expensive, and consensus is hard: you don't want to have to go to five of eight government offices in order to get your passport issued, and organizations where every decision is made by a large executive board tend to decrease quickly in efficiency. in cryptoland, however, we get to benefit from forty years of rapid development of low-cost computer hardware capable of executing billions of processing cycles per second in silicon and so, it is rational to at least explore the hypothesis that the optimal tradeoffs should be different. this is in some ways the decentralized software industry's ultimate bet now let's go ahead and see how far we can take it. the next part of the series will discuss the future of blockchain technology from a technical perspective, and show what decentralized computation and transaction processing platforms may look like in ten years' time. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements an introduction to futarchy | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search an introduction to futarchy posted by vitalik buterin on august 21, 2014 research & development one of the more interesting long-term practical benefits of the technology and concept behind decentralized autonomous organizations is that daos allow us to very quickly prototype and experiment with an aspect of our social interactions that is so far arguably falling behind our rapid advancements in information and social technology elsewhere: organizational governance. although our modern communications technology is drastically augmenting individuals' naturally limited ability to both interact and gather and process information, the governance processes we have today are still dependent on what may now be seen as centralized crutches and arbitrary distinctions such as "member", "employee", "customer" and "investor" features that were arguably originally necessary because of the inherent difficulties of managing large numbers of people up to this point, but perhaps no longer. now, it may be possible to create systems that are more fluid and generalized that take advantage of the full power law curve of people's ability and desire to contribute. there are a number of new governance models that try to take advantage of our new tools to improve transparency and efficiency, including liquid democracy and holacracy; the one that i will discuss and dissect today is futarchy. the idea behind futarchy was originally proposed by economist robin hanson as a futuristic form of government, following the slogan: vote values, but bet beliefs. under this system, individuals would vote not on whether or not to implement particular policies, but rather on a metric to determine how well their country (or charity or company) is doing, and then prediction markets would be used to pick the policies that best optimize the metric. given a proposal to approve or reject, two prediction markets would be created each containing one asset, one market corresponding to acceptance of the measure and one to rejection. if the proposal is accepted, then all trades on the rejection market would be reverted, but on the acceptance market after some time everyone would be paid some amount per token based on the futarchy's chosen success metric, and vice versa if the proposal is rejected. the market is allowed to run for some time, and then at the end the policy with the higher average token price is chosen. our interest in futarchy, as explained above, is in a slightly different form and use case of futarchy, governing decentralized autonomous organizations and cryptographic protocols; however, i am presenting the use of futarchy in a national government first because it is a more familiar context. so to see how futarchy works, let's go through an example. suppose that the success metric chosen is gdp in trillions of dollars, with a time delay of ten years, and there exists a proposed policy: "bail out the banks". two assets are released, each of which promises to pay $1 per token per trillion dollars of gdp after ten years. the markets might be allowed to run for two weeks, during which the "yes" token fetches an average price of $24.94 (meaning that the market thinks that the gdp after ten years will be $24.94 trillion) and the "no" token fetches an average price of $26.20. the banks are not bailed out. all trades on the "yes" market are reverted, and after ten years everyone holding the asset on the "no" market gets $26.20 apiece. typically, the assets in a futarchy are zero-supply assets, similar to ripple ious or bitassets. this means that the only way the tokens can be created is through a derivatives market; individuals can place orders to buy or sell tokens, and if two orders match the tokens are transferred from the buyer to the seller in exchange for usd. it's possible to sell tokens even if you do not have them; the only requirement in that case is that the seller must put down some amount of collateral to cover the eventual negative reward. an important consequence of the zero-supply property is that because the positive and negative quantities, and therefore rewards cancel each other out, barring communication and consensus costs the market is actually free to operate. the argument for futarchy has become a controversial subject since the idea was originally proposed. the theoretical benefits are numerous. first of all, futarchy fixes the "voter apathy" and "rational irrationality" problem in democracy, where individuals do not have enough incentive to even learn about potentially harmful policies because the probability that their vote will have an effect is insignificant (estimated at 1 in 10 million for a us government national election); in futarchy, if you have or obtain information that others do not have, you can personally substantially profit from it, and if you are wrong you lose money. essentially, you are literally putting your money where your mouth is. second, over time the market has an evolutionary pressure to get better; the individuals who are bad at predicting the outcome of policies will lose money, and so their influence on the market will decrease, whereas the individuals who are good at predicting the outcome of policies will see their money and influence on the market increase. note that this is essentially the exact same mechanic through which economists argue that traditional capitalism works at optimizing the production of private goods, except in this case it also applies to common and public goods. third, one could argue that futarchy reduces potentially irrational social influences to the governance process. it is a well-known fact that, at least in the 20th century, the taller presidential candidate has been much more likely to win the election (interestingly, the opposite bias existed pre-1920; a possible hypothesis is that the switchover was caused by the contemporaneous rise of television), and there is the well-known story about voters picking george bush because he was the president "they would rather have a beer with". in futarchy, the participatory governance process will perhaps encourage focusing more purely on proposals rather than personalities, and the primary activity is the most introverted and unsocial affair imaginable: poring over models, statistical analyses and trading charts. a market you would rather have a beer with the system also elegantly combines public participation and professional analysis. many people decry democracy as a descent to mediocrity and demagoguery, and prefer decisions to be made by skilled technocratic experts. futarchy, if it works, allows individual experts and even entire analysis firms to make individual investigations and analyses, incorporate their findings into the decision by buying and selling on the market, and make a profit from the differential in information between themselves and the public sort of like an information-theoretic hydroelectric dam or osmosis-based power plant. but unlike more rigidly organized and bureaucratic technocracies with a sharp distinction between member and non-member, futarchies allow anyone to participate, set up their own analysis firm, and if their analyses are successful eventually rise to the top exactly the kind of generalization and fluidity we are looking for. the argument against the opposition to futarchy is most well-summarized in two posts, one by mencius moldbug and the other by paul hewitt. both posts are long, taking up thousands of words, but the general categories of opposition can be summarized as follows: a single powerful entity or coalition wishing to see a particular result can continue buying "yes" tokens on the market and short-selling "no" tokens in order to push the token prices in its favor. markets in general are known to be volatile, and this happens to a large extent because markets are "self-referential" ie. they consist largely of people buying because they see others buying, and so they are not good aggregators of actual information. this effect is particularly dangerous because it can be exploited by market manipulation. the estimated effect of a single policy on a global metric is much smaller than the "noise" of uncertainty in what the value of the metric is going to be regardless of the policy being implemented, especially in the long term. this means that the prediction market's results may prove to be wildly uncorrellated to the actual delta that the individual policies will end up having. human values are complex, and it is hard to compress them into one numerical metric; in fact, there may be just as many disagreements about what the metric should be as there are disagreements about policy now. additionally, a malicious entity that in current democracy would try to lobby through a harmful policy might instead be able to cheat the futarchy by lobbying in an addition to the metric that is known to very highly correllate with the policy. a prediction market is zero-sum; hence, because participation has guaranteed nonzero communication costs, it is irrational to participate. thus, participation will end up quite low, so there will not be enough market depth to allow experts and analysis firms to sufficiently profit from the process of gathering information. on the first argument, this video debate between robin hanson and mencius moldbug, with david friedman (milton's son) later chiming in, is perhaps the best resource. the argument made by hanson and friedman is that the presence of an organization doing such a thing successfully would lead to a market where the prices for the "yes" and "no" tokens do not actually reflect the market's best knowledge, presenting a massive profit-earning opportunity for people to put themselves on the opposite side of the attempted manipulation and thereby move the price back closer to the correct equilibrium. in order to give time for this to happen, the price used in determining which policy to take is taken as an average over some period of time, not at one instant. as long as the market power of people willing to earn a profit by counteracting manipulation exceeds the market power of the manipulator, the honest participants will win and extract a large quantity of funds from the manipulator in the process. essentially, for hanson and friedman, sabotaging a futarchy requires a 51% attack. the most common rebuttal to this argument, made more eloquently by hewitt, is the "self-referential" property of markets mentioned above. if the price for "trillions of us gdp in ten years if we bail out the banks" starts off $24.94, and the price for "trillions of us gdp in ten years if we don't bail out the banks" starts off $26.20, but then one day the two cross over to $27.3 for yes and $25.1 for no, would people actually know that the values are off and start making trades to compensate, or would they simply take the new prices as an indicator of what the market thinks and accept or even reinforce them, as is often theorized to happen in speculative bubbles? self-reference there is actually one reason to be optimistic here. traditional markets may perhaps be often self-referential, and cryptocurrency markets especially so because they have no intrinsic value (ie. the only source of their value is their value), but the self-reference happens in part for a different reason than simply investors following each other like lemmings. the mechanism is as follows. suppose that a company is interested in raising funds through share issuance, and currently has a million shares valued at $400, so a market cap of $400 million; it is willing to dilute its holders with a 10% expansion. thus, it can raise $40 million. the market cap of the company is supposed to target the total amount of dividends that the company will ever pay out, with future dividends appropriately discounted by some interest rate; hence, if the price is stable, it means that the market expects the company to eventually release the equivalent of $400 million in total dividends in present value. now, suppose the company's share price doubles for some reason. the company can now raise $80 million, allowing it to do twice as much. usually, capital expenditure has diminishing returns, but not always; it may happen that with the extra $40 million capital the company will be able to earn twice as much profit, so the new share price will be perfectly justified even though the cause of the jump from $400 to $800 may have been manipulation or random noise. bitcoin has this effect in an especially pronounced way; when the price goes up, all bitcoin users get richer, allowing them to build more businesses, justifying the higher price level. the lack of intrinsic value for bitcoin means that the self-referential effect is the only effect having influence on the price. prediction markets do not have this property at all. aside from the prediction market itself, there is no plausible mechanism by which the price of the "yes" token on a prediction market will have any impact on the gdp of the us in ten years. hence, the only effect by which self-reference can happen is the "everyone follows everyone else's judgement" effect. however, the extent of this effect is debatable; perhaps because of the very recognition that the effect exists, there is now an established culture of smart contrarianism in investment, and politics is certainly an area where people are willing to keep to unorthodox views. additionally, in a futarchy, the relevant thing is not how high individual prices are, but which one of the two is higher; if you are certain that bailouts are bad, but you see the yes-bailout price is now $2.2 higher for some reason, you know that something is wrong so, in theory, you might be able to pretty reliably profit from that. absolutes and differentials this is where we get to the crux of the real problem: it's not clear how you can. consider a more extreme case than the yes/no bailouts decision: a company using a futarchy to determine how much to pay their ceo. there have been studies suggesting that ultra-high-salary ceos actually do not improve company performance in fact, much the opposite. in order to fix this problem, why not use the power of futarchy and the market decide how much value the ceo really provides? have a prediction market for the company's performance if the ceo stays on, and if the ceo jumps off, and take the ceo's salary as a standard percentage of the difference. we can do the same even for lower-ranking executives and if futarchy ends up being magically perfect even the lowliest employee. now, suppose that you, as an analyst, predict that a company using such a scheme will have a share price of $7.20 in twelve months if the ceo stays on, with a 95% confidence interval of $2.50 (ie. you're 95% sure the price will be between $4.70 and $9.70). you also predict that the ceo's benefit to the share price is $0.08; the 95% confidence interval that you have here is from $0.03 to $0.13. this is pretty realistic; generally errors in measuring a variable are proportional to the value of that variable, so the range on the ceo will be much lower. now suppose that the prediction market has the token price of $7.70 if the ceo stays on and $7.40 if they leave; in short, the market thinks the ceo is a rockstar, but you disagree. but how do you benefit from this? the initial instinct is to buy "no" shares and short-sell "yes" shares. but how many of each? you might think "the same number of each, to balance things out", but the problem is that the chance the ceo will remain on the job is much higher than 50%. hence, the "no" trades will probably all be reverted and the "yes" trades will not, so alongside shorting the ceo what you are also doing is taking a much larger risk shorting the company. if you knew the percentage change, then you could balance out the short and long purchases such that on net your exposure to unrelated volatility is zero; however, because you don't, the risk-to-reward ratio is very high (and even if you did, you would still be exposed to the variance of the company's global volatility; you just would not be biased in any particular direction). from this, what we can surmise is that futarchy is likely to work well for large-scale decisions, but much less well for finer-grained tasks. hence, a hybrid system may work better, where a futarchy decides on a political party every few months and that political party makes decisions. this sounds like giving total control to one party, but it's not; note that if the market is afraid of one-party control then parties could voluntarily structure themselves to be composed of multiple groups with competing ideologies and the market would prefer such combinations; in fact, we could have a system where politicians sign up as individuals and anyone from the public can submit a combination of politicians to elect into parliament and the market would make a decision over all combinations (although this would have the weakness that it is once again more personality-driven). futarchy and protocols and daos all of the above was discussing futarchy primarily as a political system for managing government, and to a lesser extent corporations and nonprofits. in government, if we apply futarchy to individual laws, especially ones with relatively small effect like "reduce the duration of patents from 20 years to 18 years", we run into many of the issues that we described above. additionally, the fourth argument against futarchy mentioned above, the complexity of values, is a particular sore point, since as described above a substantial portion of political disagreement is precisely in terms of the question of what the correct values are. between these concerns, and political slowness in general, it seems unlikely that futarchy will be implemented on a national scale any time soon. indeed, it has not even really been tried for corporations. now, however, there is an entirely new class of entities for which futarchy might be much better suited, and where it may finally shine: daos. to see how futarchy for daos might work, let us simply describe how a possible protocol would run on top of ethereum: every round, t new dao-tokens are issued. at the start of a round, anyone has the ability to make a proposal for how those coins should be distributed. we can simplify and say that a "proposal" simply consists of "send money to this address"; the actual plan for how that money would be spent would be communicated on some higher-level channel like a forum, and trust-free proposals could be made by sending to a contract. suppose that n such proposals, p[1] ... p[n], are made. the dao generates n pairs of assets, r[i] and s[i], and randomly distributes the t units of each type of token in some fashion (eg. to miners, to dao token holders, according to a formula itself determined through prior futarchy, etc). the dao also provides n markets, where market m[i] allows trade between r[i] and s[i]. the dao watches the average price of s[i] denominated in r[i] for all markets, and lets the markets run for b blocks (eg. 2 weeks). at the end of the period, if market m[k] has the highest average price, then policy p[k] is chosen, and the next period begins. at that point, tokens r[j] and s[j] for j != k become worthless. token r[k] is worth m units of some external reference asset (eg. eth for a futarchy on top of ethereum), and token s[k] is worth z dao tokens, where a good value for z might be 0.1 and m self-adjusts to keep expenditures reasonable. note that for this to work the dao would need to also sell its own tokens for the external reference asset, requiring another allocation; perhaps m should be targeted so the token expenditure to purchase the required ether is zt. essentially, what this protocol is doing is implementing a futarchy which is trying to optimize for the token's price. now, let's look at some of the differences between this kind of futarchy and futarchy-for-government. first, the futarchy here is making only a very limited kind of decision: to whom to assign the t tokens that are generated in each round. this alone makes the futarchy here much "safer". a futarchy-as-government, especially if unrestrained, has the potential to run into serious unexpected issues when combined with the fragility-of-value problem: suppose that we agree that gdp per capita, perhaps even with some offsets for health and environment, is the best value function to have. in that case, a policy that kills off the 99.9% of the population that are not super-rich would win. if we pick plain gdp, then a policy might win that extremely heavily subsidizes individuals and businesses from outside relocating themselves to be inside the country, perhaps using a 99% one-time capital tax to pay for a subsidy. of course, in reality, futarchies would patch the value function and make a new bill to reverse the original bill before implementing any such obvious egregious cases, but if such reversions become too commonplace then the futarchy essentially degrades into being a traditional democracy. here, the worst that could happen is for all the n tokens in a particular round to go to someone who will squander them. second, note the different mechanism for how the markets work. in traditional futarchy, we have a zero-total-supply asset that is traded into existence on a derivatives market, and trades on the losing market are reverted. here, we issue positive-supply assets, and the way that trades are reverted is that the entire issuance process is essentially reverted; both assets on all losing markets become worth zero. the biggest difference here is the question of whether or not people will participate. let us go back to the earlier criticism of futarchy, that it is irrational to participate because it is a zero-sum game. this is somewhat of a paradox. if you have some inside information, then you might think that it is rational to participate, because you know something that other people don't and thus your expectation of the eventual settlement price of the assets is different from the market's; hence, you should be able to profit from the difference. on the other hand, if everyone thinks this way, then even some people with inside information will lose out; hence, the correct criterion for participating is something like "you should participate if you think you have better inside information than everyone else participating". but if everyone thinks this way then the equilibrium will be that no one participates. here, things work differently. people participate by default, and it's harder to say what not participating is. you could cash out your r[i] and s[i] coins in exchange for dao tokens, but then if there's a desire to do that then r[i] and s[i] would be undervalued and there would be an incentive to buy both of them. holding only r[i] is also not non-participating; it's actually an expression of being bearish on the merits of policy p[i]; same with holding only s[i]. in fact, the closest thing to a "default" strategy is holding whatever r[i] and s[i] you get; we can model this prediction market as a zero-supply market plus this extra initial allocation, so in that sense the "just hold" approach is a default. however, we can argue that the barrier to participation is much lower, so participation will increase. also note that the optimization objective is simpler; the futarchy is not trying to mediate the rules of an entire government, it is simply trying to maximize the value of its own token by allocating a spending budget. figuring out more interesting optimization objectives, perhaps ones that penalize common harmful acts done by existing corporate entities, is an unsolved challenge but a very important one; at that point, the measurement and metric manipulation issues might once again become more important. finally, the actual day-to-day governance of the futarchy actually does follow a hybrid model; the disbursements are made once per epoch, but the management of the funds within that time can be left to individuals, centralized organizations, blockchain-based organizations or potentially other daos. thus, we can expect the differences in expected token value between the proposals to be large, so the futarchy actually will be fairly effective or at least more effective than the current preferred approach of "five developers decide". why? so what are the practical benefits of adopting such a scheme? what is wrong with simply having blockchain-based organizations that follow more traditional models of governance, or even more democratic ones? since most readers of this blog are already cryptocurrency advocates, we can simply say that the reason why this is the case is the same reason why we are interested in using cryptographic protocols instead of centrally managed systems cryptographic protocols have a much lower need for trusting central authorities (if you are not inclined to distrust central authorities, the argument can be more accurately rephrased as "cryptographic protocols can more easily generalize to gain the efficiency, equity and informational benefits of being more participatory and inclusive without leading to the consequence that you end up trusting unknown individuals"). as far as social consequences go, this simple version of futarchy is far from utopia, as it is still fairly similar to a profit-maximizing corporation; however, the two important improvements that it does make are (1) making it harder for executives managing the funds to cheat both the organization and society for their short-term interest, and (2) making governance radically open and transparent. however, up until now, one of the major sore points for a cryptographic protocol is how the protocol can fund and govern itself; the primary solution, a centralized organization with a one-time token issuance and presale, is basically a hack that generates initial funding and initial governance at the cost of initial centralization. token sales, including our own ethereum ether sale, have been a controversial topic, to a large extent because they introduce this blemish of centralization into what is otherwise a pure and decentralized cryptosystem; however, if a new protocol starts off issuing itself as a futarchy from day one, then that protocol can achieve incentivization without centralization one of the key breakthroughs in economics that make the cryptocurrency space in general worth watching. some may argue that inflationary token systems are undesirable and that dilution is bad; however, an important point is that, if futarchy works, this scheme is guaranteed to be at least as effective as a fixed-supply currency, and in the presence of a nonzero quantity of potentially satisfiable public goods it will be strictly superior. the argument is simple: it is always possible to come up with a proposal that sends the funds to an unspendable address, so any proposal that wins would have to win against that baseline as well. so what are the first protocols that we will see using futarchy? theoretically, any of the higher-level protocols that have their own coin (eg. swarm, storj, maidsafe), but without their own blockchain, could benefit from futarchy on top of ethereum. all that they would need to do is implement the futarchy in code (something which i have started to do already), add a pretty user interface for the markets, and set it going. although technically every single futarchy that starts off will be exactly the same, futarchy is schelling-point-dependent; if you create a website around one particular futarchy, label it "decentralized insurance", and gather a community around that idea, then it will be more likely that that particular futarchy succeeds if it actually follows through on the promise of decentralized insurance, and so the market will favor proposals that actually have something to do with that particular line of development. if you are building a protocol that will have a blockchain but does not yet, then you can use futarchy to manage a "protoshare" that will eventually be converted over; and if you are building a protocol with a blockchain from the start you can always include futarchy right into the core blockchain code itself; the only change will be that you will need to find something to replace the use of a "reference asset" (eg. 264 hashes may work as a trust-free economic unit of account). of course, even in this form futarchy cannot be guaranteed to work; it is only an experiment, and may well prove inferior to other mechanisms like liquid democracy or hybrid solutions may be best. but experiments are what cryptocurrency is all about. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements daos, dacs, das and more: an incomplete terminology guide | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search daos, dacs, das and more: an incomplete terminology guide posted by vitalik buterin on may 6, 2014 research & development one of the most popular topics in the digital consensus space (a new term for cryptocurrency 2.0 that i’m beta-testing) is the concept of decentralized autonomous entities. there are now a number of groups rapidly getting involved in the space, including bitshares (also known as invictus innovations) developing “decentralized autonomous companies”, bitangels’ david johnston with decentralized applications, our own concept of decentralized autonomous corporations which has since transformed into the much more general and not necessarily financial “decentralized autonomous organizations” (daos); all in all, it is safe to say that “daoism” is well on its way to becoming a quasi-cyber-religion. however, one of the hidden problems lurking beneath the space is a rather blatant one: no one even knows what all of these invididual terms mean. what exactly is a decentralized organization, what is the difference between an organization and an application, and what even makes something autonomous in the first place? many of us have been frustrated by the lack of coherent terminology here; as bitshares’ daniel larimer points out, “everyone thinks a dac is just a way of ipoing your centralized company.” the intent of this article will be to delve into some of these concepts, and see if we can come up with at least the beginnings of a coherent understanding of what all of these things actually are. smart contracts a smart contract is the simplest form of decentralized automation, and is most easily and accurately defined as follows: a smart contract is a mechanism involving digital assets and two or more parties, where some or all of the parties put assets in and assets are automatically redistributed among those parties according to a formula based on certain data that is not known at the time the contract is initiated. one example of a smart contract would be an employment agreement: a wants to pay 500tobtobuildawebsite.thecontractwouldworkasfollows:aputs500 to b to build a website. the contract would work as follows: a puts 500tobtobuildawebsite.thecontractwouldworkasfollows:aputs500 into the contract, and the funds are locked up. when b finishes the website, b can send a message to the contract asking to unlock the funds. if a agrees, the funds are released. if b decides not to finish the website, b can quit by sending a message to relinquish the funds. if b claims that he finished the website, but a does not agree, then after a 7-day waiting period it’s up to judge j to provide a verdict in a or b’s favor. the key property of a smart contract is simple: there is only a fixed number of parties. the parties do not all have to be known at initialization-time; a sell order, where a offers to sell 50 units of asset a to anyone who can provide 10 units of asset b, is also a smart contract. smart contracts can run on forever; hedging contracts and escrow contracts are good examples there. however, smart contracts that run on forever should still have a fixed number of parties (eg. an entire decentralized exchange is not a smart contract), and contracts that are not intended to exist forever are smart contracts because existing for a finite time necessarily implies the involvement of a finite number of parties. note that there is one gray area here: contracts which are finite on one side, but infinite on the other side. for example, if i want to hedge the value of my digital assets, i might want to create a contract where anyone can freely enter and leave. hence, the other side of the contract, the parties that are speculating on the asset at 2x leverage, has an unbounded number of parties, but my side of the contract does not. here, i propose the following divide: if the side with a bounded number of parties is the side that intends to receive a specific service (ie. is a consumer), then it is a smart contract; however, if the side with a bounded number of parties is just in it for profit (ie. is a producer), then it is not. autonomous agents autonomous agents are on the other side of the automation spectrum; in an autonomous agent, there is no necessary specific human involvement at all; that is to say, while some degree of human effort might be necessary to build the hardware that the agent runs on, there is no need for any humans to exist that are aware of the agent’s existence. one example of an autonomous agent that already exists today would be a computer virus; the virus survives by replicating itself from machine to machine without deliberate human action, and exists almost as a biological organism. a more benign entity would be a decentralized self-replicating cloud computing service; such a system would start off running an automated business on one virtual private server, and then once its profits increase it would rent other servers and install its own software on them, adding them to its network. a full autonomous agent, or a full artificial intelligence, is the dream of science fiction; such an entity would be able to adjust to arbitrary changes in circumstances, and even expand to manufacture the hardware needed for its own sustainability in theory. between that, and single purpose agents like computer viruses, is a large range of possibilities, on a scale which can alternatively be described as intelligence or versatility. for example, the self-replicating cloud service, in its simplest form, would only be able to rent servers from a specific set of providers (eg. amazon, microtronix and namecheap). a more complex version, however, should be able to figure out how to rent a server from any provider given only a link to its website, and then use any search engine to locate new websites (and, of course, new search engines in case google fails). the next level from there would involve upgrading its own software, perhaps using evolutionary algorithms, or being able to adapt to new paradigms of server rental (eg. make offers for ordinary users to install its software and earn funds with their desktops), and then the penultimate step consists of being able to discover and enter new industries (the ultimate step, of course, is generalizing completely into a full ai). autonomous agents are some of the hardest things to create, because in order to be successful they need to be able to navigate in an environment that is not just complicated and rapidly changing, but also hostile. if a web hosting provider wants to be unscrupulous, they might specifically locate all instances of the service, and then replace them with nodes that cheat in some fashion; an autonomous agent must be able to detect such cheating and remove or at least neutralize cheating nodes from the system. decentralized applications a decentralized application is similar to a smart contract, but different in two key ways. first of all, a decentralized application has an unbounded number of participants on all sides of the market. second, a decentralized application need not be necessarily financial. because of this second requirement, decentralized applications are actually some of the easiest things to write (or at least, were the easiest before generalized digital consensus platforms came along). for example, bittorrent qualifies as a decentralized application, as do popcorn time, bitmessage, tor and maidsafe (note that maidsafe is also itself a platform for other decentralized applications). generally, decentralized applications fall into two classes, likely with a substantial gray area between the two. the first class is a fully anonymous decentralized application. here, it does not matter who the nodes are; every participant is essentially anonymous and the system is made up of a series of instant atomic interactions. bittorrent and bitmessage are examples of this. the second class is a reputation-based decentralized application, where the system (or at least nodes in the system) keep track of nodes, and nodes maintain status inside of the application with a mechanism that is purely maintained for the purpose of ensuring trust. status should not be transferable or have de-facto monetary value. maidsafe is an example of this. of course, purity is impossible – even a bittorrent-like system needs to have peers maintain reputation-like statistics of other peers for anti-ddos purposes; however, the role that these statistics play is purely in the background and very limited in scope. an interesting gray area between decentralized applications and “something else” is applications like bitcoin and namecoin; these differ from traditional applications because they create ecosystems and there is a concept of virtual property that has value inside the context of this ecosystem, in bitcoin’s case bitcoins and in namecoin’s case namecoins and domain names. as we’ll see below, my classification of decentralized autonomous organizations touches on such concepts, and it is not quite clear exactly where they sit. decentralized organizations in general, a human organization can be defined as combination of two things: a set of property, and a protocol for a set of individuals, which may or may not be divided into certain classes with different conditions for entering or leaving the set, to interact with each other including rules for under what circumstances the individuals may use certain parts of the property. for example, consider a simple corporation running a chain of stores. the corporation has three classes of members: investors, employees and customers. the membership rule for investors is that of a fixed-size (or optionally quorum-adjustable size) slice of virtual property; you buy some virtual property to get in, and you become an investor until you sell your shares. employees need to be hired by either investors or other employees specifically authorized by investors (or other employees authorized by other employees authorized by investors, and so on recursively) to participate, and can also be fired in the same way, and customers are an open-membership system where anyone can freely interact with the store in the obvious officially sanctioned way for any time. suppliers, in this model, are equivalent to employees. a nonprofit charity has a somewhat different structure, involving donors and members (charity recipients may or may not be considered members; the alternative view sees the positive increments in the recipients’ welfare as being the charity’s “product”). the idea of a decentralized organization takes the same concept of an organization, and decentralizes it. instead of a hierarchical structure managed by a set of humans interacting in person and controlling property via the legal system, a decentralized organization involves a set of humans interacting with each other according to a protocol specified in code, and enforced on the blockchain. a do may or may not make use of the legal system for some protection of its physical property, but even there such usage is secondary. for example, one can take the shareholder-owned corporation above, and transplant it entirely on the blockchain; a long-running blockchain-based contract maintains a record of each individual’s holdings of their shares, and on-blockchain voting would allow the shareholders to select the positions of the board of directors and the employees. smart property systems can also be integrated into the blockchain directly, potentially allowing dos to control vehicles, safety deposit boxes and buildings. decentralized autonomous organizations here, we get into what is perhaps the holy grail, the thing that has the murkiest definition of all: decentralized autonomous organizations, and their corporate subclass, decentralized autonomous corporations (or, more recently, “companies”). the ideal of a decentralized autonomous organization is easy to describe: it is an entity that lives on the internet and exists autonomously, but also heavily relies on hiring individuals to perform certain tasks that the automaton itself cannot do. given the above, the important part of the definition is actually to focus on what a dao is not, and what is not a dao and is instead either a do, a da or an automated agent/ai. first of all, let’s consider das. the main difference between a da and a dao is that a dao has internal capital; that is, a dao contains some kind of internal property that is valuable in some way, and it has the ability to use that property as a mechanism for rewarding certain activities. bittorrent has no internal property, and bitcloud/maidsafe-like systems have reputation but that reputation is not a saleable asset. bitcoin and namecoin, on the other hand, do. however, plain old dos also have internal capital, as do autonomous agents. second, we can look at dos. the obvious difference between a do and a dao, and the one inherent in the language, is the word “autonomous”; that is, in a do the humans are the ones making the decisions, and a dao is something that, in some fashion, makes decisions for itself. this is a surprisingly tricky distinction to define because, as dictatorships are always keen to point out, there is really no difference between a certain set of actors making decisions directly and that set of actors controlling all of the information through which decisions are made. in bitcoin, a 51% attack between a small number of mining pools can make the blockchain reverse transactions, and in a hypothetical decentralized autonomous corporation the providers of the data inputs can all collude to make the dac think that sending all of its money to1fxkfjqljtxpw6qmxgt6of43zh959ns8cq constitutes paying for a million nodes’ worth of computing power for ten years. however, there is obviously a meaningful distinction between the two, and so we do need to define it. my own effort at defining the difference is as follows. dos and daos are both vulnerable to collusion attacks, where (in the best case) a majority or (in worse cases) a significant percentage of a certain type of members collude to specifically direct the d*o’s activity. however, the difference is this: in a dao collusion attacks are treated as a bug, whereas in a do they are a feature. in a democracy, for example, the whole point is that a plurality of members choose what they like best and that solution gets executed; in bitcoin’s on the other hand, the “default” behavior that happens when everyone acts according to individual interest without any desire for a specific outcome is the intent, and a 51% attack to favor a specific blockchain is an aberration. this appeal to social consensus is similar to the definition of a government: if a local gang starts charging a property tax to all shopowners, it may even get away with it in certain parts of the world, but no significant portion of the population will treat it as legitimate, whereas if a government starts doing the same the public response will be tilted in the other direction. bitcoin is an interesting case here. in general, it seems to be much closer to a dao than a do. however, there was one incident in 2013 where the reality proved to be rather different. what happened was that an exceptional block was (at least we hope) accidentally produced, which was treated as valid according to the bitcoinqt 0.8 clients, but invalid according to the rules of bitcoinqt 0.7. the blockchain forked, with some nodes following the blockchain after this exceptional block (we’ll call this chain b1), and the other nodes that saw that block as invalid working on a separate blockchain (which we’ll call b2). most mining pools had upgraded to bitcoinqt 0.8, so they followed b1, but most users were still on 0.7 and so followed b2. the mining pool operators came together on irc chat, and agreed to switch their pools to mining on b2, since that outcome would be simpler for users because it would not require them to upgrade, and after six hours the b2 chain overtook b1 as a result of this deliberate action, and b1 fell away. thus, in this case, there was a deliberate 51% attack which was seen by the community as legitimate, making bitcoin a do rather than a dao. in most cases, however, this does not happen, so the best way to classify bitcoin would be as a dao with an imperfection in its implementation of autonomy. however, others are not content to classify bitcoin as a dao, because it is not really smart enough. bitcoin does not think, it does not go out and “hire” people with the exception of the mining protocol, and it follows simple rules the upgrading process for which is more do-like than dao-like. people with this view would see a dao as something that has a large degree of autonomous intelligence of its own. however, the issue with this view is that there must be a distinction made between a dao and an aa/ai. the distinction here is arguably this: an ai is completely autonomous, whereas a dao still requires heavy involvement from humans specifically interacting according to a protocol defined by the dao in order to operate. we can classify daos, dos (and plain old os), ais and a fourth category, plain old robots, according to a good old quadrant chart, with another quadrant chart to classify entities that do not have internal capital thus altogether making a cube: daos == automation at the center, humans at the edges. thus, on the whole, it makes most sense to see bitcoin and namecoin as daos, albeit ones that barely cross the threshold from the da mark. the other important distinction is internal capital; a dao without internal capital is a da and an organization without internal capital is a forum; the g8, for example, would qualify as a forum. dcs in the graph above are “decentralized communities”; an example of that might be something like a decentralized reddit, where there is a decentralized platform, but there is also a community around that platform, and it is somewhat ambiguous whether the community or the protocol is truly “in charge”. decentralized autonomous corporations decentralized autonomous corporations/companies are a smaller topic, because they are basically a subclass of daos, but they are worth mentioning. since the main exponent of dac as terminology is daniel larimer, we will borrow as a definition the point that he himself consistently promotes: a dac pays dividends. that is, there is a concept of shares in a dac which are purchaseable and tradeable in some fashion, and those shares potentially entitle their holders to continual receipts based on the dac’s success. a dao is non-profit; though you can make money in a dao, the way to do that is by participating in its ecosystem and not by providing investment into the dao itself. obviously, this distinction is a murky one; all daos contain internal capital that can be owned, and the value of that internal capital can easily go up as the dao becomes more powerful/popular, so a large portion of daos are inevitably going to be dac-like to some extent. thus, the distinction is more of a fluid one and hinges on emphasis: to what extent are dividends the main point, and to what extent is it about earning tokens by participation? also, to what extent does the concept of a “share” exist as opposed to simple virtual property? for example, a membership on a nonprofit board is not really a share, because membership frequently gets granted and confiscated at will, something which would be unacceptable for something classified as investable property, and a bitcoin is not a share because a bitcoin does not entitle you to any claim on profits or decision-making ability inside the system, whereas a share in a corporation definitely is a share. in the end, perhaps the distinction might ultimately be the surprisingly obscure point of whether or not the profit mechanism and the consensus mechanism are the same thing. the above definitions are still not close to complete; there will likely be gray areas and holes in them, and exactly what kind of automation a do must have before it becomes a dao is a very hard question to answer. additionally, there is also the question of how all of these things should be built. an ai, for example, should likely exist as a network of private servers, each one running often proprietary local code, whereas a do should be fully open source and blockchain-based. between those two extremes, there is a large number of different paradigms to pursue. how much of the intelligence should be in the core code? should genetic algorithms be used for updating code, or should it be futarchy or some voting or vetting mechanism based on individuals? should membership be corporate-style, with sellable and transferable shares, or nonprofit-style, where members can vote other members in and out? should blockchains be proof of work, proof of stake, or reputation-based? should daos try to maintain balances in other currencies, or should they only reward behavior by issuing their own internal token? these are all hard problems and we have only just begun scratching the surface of them. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements software and bounded rationality | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search software and bounded rationality posted by vitalik buterin on september 2, 2014 research & development one of the key properties that is usually sought for in a cryptoeconomic algorithm, whether a blockchain consensus algorithm such a proof of work or proof of stake, a reputation system or a trading process for something like data transmission or file storage, is the ideal of incentive-compatibility the idea that it should be in everyone's economic interest to honestly follow the protocol. the key underlying assumption in this goal is the idea that people (or more precisely in this case nodes) are "rational" that is to say, that people have a relatively simple defined set of objectives and follow the optimal strategy to maximize their achievement of those objectives. in game-theoretic protocol design, this is usually simplified to saying that people like money, since money is the one thing that can be used to help further one's success in almost any objective. in reality, however, this is not precisely the case. humans, and even the de-facto human-machine hybrids that are the participants of protocols like bitcoin and ethereum, are not perfectly rational, and there are specific deviations from rationality that are so prevalent among users that they cannot be simply categorized as "noise". in the social sciences, economics has responded to this concern with the subfield of behavioral economics, which combines experimental studies with a set of new theoretical concepts including prospect theory, bounded rationality, defaults and heuristics, and has succeeded in creating a model which in some cases considerably more accurately models human behavior. in the context of cryptographic protocols, rationality-based analyses are arguably similarly suboptimal, and there are particular parallels between some of the concepts; for example, as we will later see, "software" and "heuristic" are essentially synonyms. another point of interest is the fact that we arguably do not even have an accurate model of what constitutes an "agent", an insight that has particular importance to protocols that try to be "trust-free" or have "no single point of failure". traditional models in traditional fault-tolerance theory, there are three kinds of models that are used for determining how well a decentralized system can survive parts of it deviating from the protocol, whether due to malice or simple failure. the first of these is simple fault tolerance. in a simple fault tolerant system, the idea is that all parts of the system can be trusted to do either one of two things: exactly follow the protocol, or fail. the system should be designed to detect failures and recover and route around them in some fashion. simple fault tolerance is usually the best model for evaluating systems that are politically centralized, but architecturally decentralized; for example, amazon or google's cloud hosting. the system should definitely be able to handle one server going offline, but the designers do not need to think about one of the servers becoming evil (if that does happen, then an outage is acceptable until the amazon or google team manually figure out what is going on and shut that server down). however, simple fault tolerance is not useful for describing systems that are not just architecturally, but also politically, decentralized. what if we have a system where we want to be fault-tolerant against some parts of the system misacting, but the parts of the system might be managed by different organizations or individuals, and you do not trust all of them not to be malicious (although you do trust that at least, say, two thirds of them will act honestly)? in this case, the model we want is byzantine fault tolerance (named after the byzantine generals problem) most nodes will honestly follow the protocol, but some will deviate, and they can deviate in any way; the assumption is that all deviating nodes are colluding to screw you over. a byzantine-fault-tolerant protocol should survive against a limited number of such deviations. for an example of simple and byzantine fault-tolerance in action, a good use case is decentralized file storage. beyond these two scenarios, there is also another even more sophisticated model: the byzantine/altruistic/rational model. the bar model improves upon the byzantine model by adding a simple realization: in real life, there is no sharp distinction between "honest" and "dishonest" people; everyone is motivated by incentives, and if the incentives are high enough then even the majority of participants may well act dishonestly particularly if the protocol in question weights people's influence by economic power, as pretty much all protocols do in the blockchain space. thus, the bar model assumes three types of actors: altruistic altruistic actors always follow the protocol rational rational actors follow the protocol if it suits them, and do not follow the protocol if it does not byzantine byzantine actors are all conspiring to screw you over in practice, protocol developers tend to be uncomfortable assuming any specific nonzero quantity of altruism, so the model that many protocols are judged by is the even harsher "br" model; protocols that survive under br are said to be incentive-compatible (anything that survives under br survives under bar, since an altruist is guaranteed to be at least as good for the health of the protocol as anyone else as benefitting the protocol is their explicit objective). note that these are worst-case scenarios that the system must survive, not accurate descriptions of reality at all times to see how this model works, let us examine an argument for why bitcoin is incentive-compatible. the part of bitcoin that we care most about is the mining protocol, with miners being the users. the "correct" strategy defined in the protocol is to always mine on the block with the highest "score", where score is roughly defined as follows: if a block is the genesis block, score(b) = 0 if a block is invalid, score(b) = -infinity otherwise, score(b) = score(b.parent) + 1 in practice, the contribution that each block makes to the total score varies with difficulty, but we can ignore such subtleties in our simple analysis. if a block is successfully mined, then the miner receives a reward of 50 btc. in this case, we can see that there are exactly three byzantine strategies: not mining at all mining on a block other than the block with highest score trying to produce an invalid block the argument against (1) is simple: if you don't mine, you don't get the reward. now, let's look at (2) and (3). if you follow the correct strategy, you have a probability p of producing a valid block with score s + 1 for some s. if you follow a byzantine strategy, you have a probability p of producing a valid block with score q + 1 with q < s (and if you try to produce an invalid block, you have a probability of producing some block with score negative infinity). thus, your block is not going to be the block with the highest score, so other miners are not going to mine on it, so your mining reward will not be part of the eventual longest chain. note that this argument does not depend on altruism; it only depends on the idea that you have an incentive to keep in line if everyone else does a classic schelling point argument. the best strategy to maximize the chance that your block will get included in the eventual winning blockchain is to mine on the block that has the highest score. trust-free systems another important category of cryptoeconomic protocols is the set of so-called "trust-free" centralized protocols. of these, there are a few major categories: provably fair gambling one of the big problems in online lotteries and gambling sites is the possibility of operator fraud, where the operator of the site would slightly and imperceptibly "load the dice" in their favor. a major benefit of cryptocurrency is its ability to remove this problem by constructing a gambling protocol that is auditable, so any such deviation can be very quickly detected. a rough outline of a provably fair gambling protocol is as follows: at the beginning of each day, the site generates a seed s and publishes h(s) where h is some standard hash function (eg. sha3) when a user sends a transaction to make a bet, the "dice roll" is calculated using h(s + tx) mod n where tx is the transaction used to pay for the bet and n is the number of possible outcomes (eg. if it's a 6-sided die, n = 6, for a lottery with a 1 in 927 chance of winning, n = 927 and winning games are games where h(s + tx) mod 927 = 0). at the end of the day, the site publishes s. users can then verify that (1) the hash provided at the beginning of the day actually is h(s), and (2) that the results of the bets actually match the formulas. thus, a gambling site following this protocol has no way of cheating without getting caught within 24 hours; as soon as it generates s and needs to publish a value h(s) it is basically bound to follow the precise protocol correctly. proof of solvency another application of cryptography is the concept of creating auditable financial services (technically, gambling is a financial service, but here we are interested in services that hold your money, not just briefly manipulate it). there are strong theoretical arguments and empirical evidence that financial services of that sort are much more likely to try to cheat their users; perhaps the most parcticularly jarring example is the case of mtgox, a bitcoin exchange which shut down with over 600,000 btc of customer funds missing. the idea behind proof of solvency is as follows. suppose there is an exchange with users u[1] ... u[n] where user u[i] has balance b[i]. the sum of all balances is b. the exchange wants to prove that it actually has the bitcoins to cover everyone's balances. this is a two-part problem: the exchange must simultaneously prove that for some b it is true that (1) the sum of users' balances is b, and (ii) the exchange is in possession of at least b btc. the second is easy to prove; just sign a message with the private key that holds the bitcoins at the time. the simplest way to prove the first is to just publish everyone's balances, and let people check that their balances match the public values, but this compromises privacy; hence, a better strategy is needed. the solution involves, as usual, a merkle tree except in this case it's a funky enhanced sort of merkle tree called a "merkle sum tree". instead of each node simply being the hash of its children, every node contains the hash of its children and the sum of the values of its children: the values at the bottom are mappings of account ids to balances. the service publishes the root of the tree, and if a user wants a proof that their account is correctly included in the tree, the service can simply give them the branch of the tree corresponding to their account: there are two ways that the site can cheat, and try to get away with having a fractional reserve. first, it can try to have one of the nodes in the merkle tree incorrectly sum the values of its children. in this case, as soon as a user requests a branch containing that node they will know that something is wrong. second, it can try to insert negative values into the leaves of the tree. however, if it does this, then unless the site provides fake positive and negative nodes that cancel each other out (thus defeating the whole point), then there will be at least one legitimate user whose merkle branch will contain the negative value; in general, getting away with having x percent less than the required reserve requires counting on a specific x percent of users never performing the audit procedure a result that is actually the best that any protocol can do, given that an exchange can always simply zero out some percentage of its users' account balances if it knows that they will never discover the fraud. multisig a third application, and a very important one, is multisig, or more generally the concept of multi-key authorization. instead of your account being controlled by one private key which may get hacked, there are three keys, of which two are needed to access the account (or some other configuration, perhaps involving withdrawal limits or time-locked withdrawals; bitcoin does not support such features but more advanced systems do). the way multisig is usually implemented so far is as a 2-of-3: you have one key, the server has one key, and you have a third backup key in a safe place. in the course of normal activity, when you sign a transaction you generally sign it with your key locally, then send it to the server. the server performs some second verification process perhaps consisting of sending a confirmation code to your phone, and if it confirms that you meant to send the transaction then it signs it as well. the idea is that such a system is tolerant against any single fault, including any single byzantine fault. if you lose your password, you have a backup, which together with the server can recover your funds, and if your password is hacked, the attacker only has one password; likewise for loss or theft of the backup. if the service disappears, you have two keys. if the service is hacked or turns out to be evil, it only has one. the probability of two failures happening at the same time is very small; arguably, you are more likely to die. fundamental units all of the above arguments make one key assumption that seems trivial, but actually needs to be challenged much more closely: that the fundamental unit of the system is the computer. each node has the incentive to mine on the block with the highest score and not follow some deviant strategy. if the server gets hacked in a multisig then your computer and your backup still have 2 out of 3 keys, so you are still safe. the problem with the approach is that it implicitly assumes that users have full control over their computers, and that the users fully understand cryptography and are manually verifying the merkle tree branches. in reality, this is not the case; in fact, the very necessity of multisig in any incarnation at all is proof of this, as it acknowledges that users' computers can get hacked a replica of the behavioral-economics idea that individuals can be viewed as not being in full control of themselves. a more accurate model is to view a node as a combination of two categories of agents: a user, and one or more software providers. users in nearly all cases do not verify their software; even in my own case, even though i verify every transaction that comes out of the ethereum exodus address, using the pybitcointools toolkit that i wrote from scratch myself (others have provided patches, but even those i reviewed personally), i am still trusting that (1) the implementations of python and ubuntu that i downloaded are legitimate, and (2) that the hardware is not somehow bugged. hence, these software providers should be treated as separate entities, and their goals and incentives should be analyzed as actors in their own right. meanwhile, users should also be viewed as agents, but as agents who have limited technical capability, and whose choice set often simply consists of which software packages to install, and not precisely which protocol rules to follow. the first, and most important, observation is that the concepts of "byzantine fault tolerance" and "single point of failure" should be viewed in light of such a distinction. in theory, multisig removes all single points of failure from the cryptographic token management process. in practice, however, that is not the way that multisig is usually presented. right now, most mainstream multisig wallets are web applications, and the entity providing the web application is the same entity that manages the backup signing key. what this means is that, if the wallet provider does get hacked or does turn out to be evil, they actually have control over two out of three keys they already have the first one, and can easily grab the second one simply by making a small change to the client-side browser application they send to you every time you load the webpage. in multisig wallet providers' defense, services like bitgo and greenaddress do offer an api, allowing developers to use their key management functionality without their interface so that the two providers can be separate entities. however, the importance of this kind of separation is currently drastically underemphasized. this insight applies equally well to provably fair gambling and proof of solvency. particular, such provably fair protocols should have standard implementations, with open-source applications that can verify proofs in a standard format and in a way that is easy to use. services like exchanges should then follow those protocols, and deliver proofs which can be verifies by these external tools. if a service releases a proof that can only be verified by its own internal tools, that is not much better than no proof at all slightly better, since there is a chance that cheating will still be detected, but not by much. software, users and protocols if we actually do have two classes of entities, it will be helpful to provide at least a rough model of their incentives, so that we may better understand how they are likely to act. in general, from software providers we can roughly expect the following goals: maximize profit in the heyday of proprietary software licensing, this goal was actually easy to understand: software companies maximize their profits by having as many users as possible. the drive toward open-source and free-to-use software more recently has very many advantages, but one disadvantage is that it now makes the profit-maximization analysis much more difficult. now, software companies generally make money through commercial value-adds, the defensibility of which typically involves creating proprietary walled-garden ecosystems. even still, however, making one's software as useful as possible usually helps, at least when it doesn't interfere with a proprietary value-add. altruism altruists write software to help people, or to help realize some vision of the world. maximize reputation these days, writing open-source software is often used as a way of building up one's resume, so as to (1) appear more attractive to employers and (2) gain the social connections to maximize potential future opportunities. corporations can also do this, writing free tools to drive people to their website in order to promote other tools. laziness software providers will not write code if they can help it. the main consequence of this will be an underinvestment in features that do not benefit their users, but benefit the ecosystem like responding to requests for data unless the software ecosystem is an oligopoly. not going to jail this entails compliance with laws, which sometimes involves anti-features such as requiring identity verification, but the dominant effect of this motive is a disincentive against screwing one's customers over too blatantly (eg. stealing their funds). users we will not analyze in terms of goals but rather in terms of a behavioral model: users select software packages from an available set, download the software, and choose options from inside that software. guiding factors in software selection include: functionality what is the utility (that's the economics jargon "utility") can they derive from the options that the software provides? ease of use of particular importance is the question of how quickly they can get up and running doing what they need to do. perceived legitimacy users are more likely to download software from trustworthy or at least trustworthy-seeming entities. salience if a software package is mentioned more often, users will be more likely to go for it. an immediate consequence is that the "official" version of a software package has a large advantage over any forks. moral and ideological considerations users might prefer open source software for its own sake, reject purely parasitic forks, etc. once users download a piece of software, the main bias that we can count on is that users will stick to defaults even when it might not benefit them to; beyond that, we have more traditional biases such as loss aversion, which we will discuss briefly later. now, let us show an example of how this process works in action: bittorrent. in the bittorrent protocol, users can download files from each other a packet at a time in a decentralized fashion, but in order for one user to download a file there must be someone uploading ("seeding") it and that activity is not incentivized. in fact, it carries non-negligible costs: bandwidth consumption, cpu resource consumption, copyright-related legal risk (including risk of getting one's internet connection shut down by one's isp, or perhaps even a possibility of lawsuit). and yet people still seed vastly insufficiently, but they do. why? the situation is explained perfectly by the two-layer model: software providers want to make their software more useful, so they include the seeding functionality by default, and users are too lazy to turn it off (and some users are deliberately altruistic, though the order-of-magnitude mismatch between willingness to torrent copyrighted content and willingness to donate to artists does suggest that most participants don't really care). message-sending in bitcoin (ie. to data requests like getblockheader and getrawtransaction) is also altruistic but also similarly explainable, as is the inconsistency between transaction fees and what the economics suggest transaction fees currently should be. another example is proof of stake algorithms. proof of stake algorithms have the (mostly) common vulnerability that there is "nothing at stake" that is to say, that the default behavior in the event of a fork is to try to vote on all chains, so an attacker need only overpower all altruists that vote on one chain only, and not all altruists plus all rational actors as in the case of proof of work. here, once again we can see that this does not mean that proof of stake is completely broken. if the stake is largely controlled by a smaller number of sophisticated parties, then those parties will have their ownership in the currency as the incentive not to participate in forks, and if the stake is controlled by very many more ordinary people then there would need to be some deliberately evil software provider who would take an effort to include a multi-voting feature, and advertise it so that potentially users actually know about the feature. however, if the stake is held in custodial wallets (eg. coinbase, xapo, etc) which do not legally own the coins, but are specialized professional entities, then this argument breaks down: they have the technical ability to multi-vote, and low incentive not to, particularly if their businesses are not "bitcoin-centric" (or ethereum-centric, or ripple-cetric) and support many protocols. there is even a probabilistic multi-voting strategy which such custodial entities can use to get 99% of the benefits of multi-voting without the risk of getting caught. hence, effective proof of stake to a moderate extent depends on technologies that allow users to safely keep control of their own coins. darker consequences what we get out of the default effect is essentially a certain level of centralization, having a beneficial role by setting users' default behavior toward a socially beneficial action and thereby correcting for what would otherwise be a market failure. now, if software introduces some benefits of centralization, we can also expect some of the negative effects of centralization as well. one particular example is fragility. theoretically, bitcoin mining is an m-of-n protocol where n is in the thousands; if you do the combinatoric math, the probability that even 5% of the nodes will deviate from the protocol is infinitesimally small, so bitcoin should have pretty much perfect reliability. in reality, of course, this is incorrect; bitcoin has had no less than two outages in the last six years. for those who do not remember, the two cases were as follows: driver of 43-year-old car exploits integer overflow vulnerability, sells it for 91% of original purchase price passing it off as new in 2010, an unknown user created a transaction with two outputs, each containing slightly more than 263 satoshis. the two outputs combined were slightly over 264, and integer overflow led to the total wrapping around to near-zero, causing the bitcoin client to think that the transaction actually released only the same small quantity of btc that it consumed as an input, and so was legitimate. the bug was fixed, and the blockchain reverted, after nine hours. in 2013, a new version of the bitcoin client unknowingly fixed a bug in which a block that made over 5000 accesses to a certain database resource would cause a berkeleydb error, leading to the client rejecting the block. such a block soon appeared, and new clients accepted it and old clients rejected it, leading to a fork. the fork was fixed in six hours, but in the meantime $10000 of btc was stolen from a payment service provider in a double-spend attack. in both cases, the network was only able to fail because, even though there were thousands of nodes, there was only one software implementation running them all perhaps the ultimate fragility in a network that is often touted for being antifragile. alternative implementations such as btcd are now increasingly being used, but it will be years before bitcoin core's monopoly is anything close to broken; and even then fragility will still be fairly high. endowment effects and defaults an important set of biases to keep in mind on the user side are the concepts of the endowment effect, loss aversion, and the default effect. the three often go hand in hand, but are somewhat different from each other. the default effect is generally most accurately modeled as a tendency to continue following one's current strategy unless there is a substantial benefit to switching in essence, an artificial psychological switching cost of some value ε. the endowment effect is the tendency to see things as being more valuable if one already has them, and loss aversion is the tendency to care more about avoiding losses than seeking gains experimentally, the scaling factor seems to be consistently around 2x. the consequences of these effects pronounce themselves most strongly in the context of multi-currency environments. as one example, consider the case of employees being paid in btc. we can see that when people are paid in btc, they are much more likely to hold on to those btc than they would have been likely to buy the btc had they been paid usd; the reason is partially the default effect, and partially the fact that if someone is paid in btc they "think in btc" so if they sell to usd then if the value of btc goes up after that they have a risk of suffering a loss, whereas if someone is paid in usd it is the usd-value of their btc that they are more concerned with. this applies also to smaller token systems; if you pay someone in zetacoin, they are likely to cash out into btc or some other coin, but the probability is much less than 100%. the loss aversion and default effects are some of the strongest arguments in favor of the thesis that a highly polycentric currency system is likely to continue to survive, contra daniel krawisz's viewpoint that btc is the one token to rule them all. there is clearly an incentive for software developers to create their own coin even if the protocol could work just as well on top of an existing currency: you can do a token sale. storj is the latest example of this. however, as daniel krawisz argues, one could simply fork such an "app-coin" and release a version on top of bitcoin, which would theoretically be superior because bitcoin is a more liquid asset to store one's funds in. the reason why such an outcome has a large chance of not happening is simply the fact that users follow defaults, and by default users will use storj with storjcoin since that is what the client will promote, and the original storj client and website and ecosystem is the one that will get all the attention. now, this argument breaks down somewhat in one case: if the fork is itself backed by a powerful entity. the latest example of this is the case of ripple and stellar; although stellar is a fork of ripple, it is backed by a large company, stripe, so the fact that the original version of a software package has the advantage of much greater salience does not apply quite as strongly. in such cases, we do not really know what will happen; perhaps, as is often the case in the social sciences, we will simply have to wait for empirical evidence to find out. the way forward relying on specific psychological features of humans in cryptographic protocol design is a dangerous game. the reason why it is good in economics to keep one's model simple, and in cryptoeconomics even more so, is that even if desires like the desire to acquire more currency units do not accurately describe the whole of human motivation, they describe an evidently very powerful component of it, and some may argue the only powerful component we can count on. in the future, education may begin to deliberately attack what we know as psychological irregularities (in fact, it already does), changing culture may lead to changing morals and ideals, and particularly in this case the agents we are dealing with are "fyborgs" functional cyborgs, or humans who have all of their actions mediated by machines like the one that sits between them and the internet. however, there are certain fundamental features of this model the concept of cryptoeconomic systems as two-layer systems featuring software and users as agents, the preference for simplicity, etc, that perhaps can be counted on, and at the very least we should try to be aware of circumstances where our protocol is secure under the bar model, but insecure under the model where a few centralized parties are in practice mediating everyone's access to the system. the model also highlights the importance of "software politics" having an understanding of the pressures that drive software development, and attempting to come up with approaches to development that software developers have the best possible incentives (or, ultimately, write software that is most favorable to the protocol's successful execution). these are problems that bitcoin has not solved, and that ethereum has not solved; perhaps some future system will do at least somewhat better. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements advanced contract programming example: schellingcoin | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search advanced contract programming example: schellingcoin posted by vitalik buterin on june 30, 2014 research & development writing effective decentralized applications in ethereum is at the same time easy and hard. the easy part we all know: rather than needing to create your own blockchain, manage complicated database code, deal with networking and nat traversal, or any of the other complexities involving writing a peer-to-peer app from scratch, you can write code in a simple, high-level programming language like serpent or mutan (or lll if you prefer mucking around a bit lower-level), with the simplicity of a toy scripting language but the power and security of a full blockchain backing it up. an entire implementation of a basic name registry can be done in two lines of code that encompass the essential logic of the program: if not contract.storage[msg.data[0]]: contract.storage[msg.data[0]] = msg.data[1]. use the zeroth data item in the message as a key and the first as a value; if the key is not yet taken then set the key to the desired value. a phone book that you can add entries to, but where entries, once made, cannot be changed. however, there is also a hard part: decentralized applications are likely to involve logic that is fundamentally complex, and there is no way that any simplifications to the programming environment can ever remove that fact (however, libraries built on top of the programming language might alleviate specific issues). furthermore, any dapps doing anything truly interesting is likely to involve cryptographic protocols and economics, and we all know how complex those are. the purpose of this article will be to go through a contract that is an important component of a fully decentralized cryptoeconomic ecosystem: a decentralized oracle. the oracle will be implemented using the schellingcoin protocol, described in a previous blog post. the core idea behind the protocol is that everyone "votes" on a particular value (in this case, we'll use wei per us cent as an example, as that will end up very useful in financial contracts), and everyone who submitted a vote that is between the 25th and 75 percentile (ie. close to median) receives a reward. the median is taken to be the "true value". in order to increase security, each round is done via a two-step commitment protocol: in the first phase, everyone selects a value p which is the value they will be voting for, and submits h = sha3([msg.sender, p]) to the contract, and in the second phase everyone submits the p that they selected and the contract accepts only those values that match the previously provided hash. rewarding and evaluation is then done at the end. the reason why it works is this. during the first phase, everyone is so to speak "in the dark"; they do not know what the others will be submitting, seeing perhaps only hashes of other votes. the only information they have is that they are supposed to be submitting the price of a us cent in wei. thus, knowing only that the only value that other people's answers are going to be biased towards is the actual wei/uscent, the rational choice to vote for in order to maximize one's chance of being near-median is the wei/uscent itself. hence, it's in everyone's best interests to come together and all provide their best estimate of the wei/uscent price. an interesting philosophical point is that this is also the same way that proof-of-work blockchains work, except that in that case what you are voting on is the time order of transactions instead of some particular numeric value; this moderately strongly suggests that this protocol is likely to be viable at least for some applications. of course, in reality various kinds of special scenarios and attacks are possible, and the fact that the price of any asset is quite often controlled by a small number of centralized exchanges makes things more difficult. for example, one imaginable failure mode is that if there is a market share split between the btc/usd on bitstamp, bitfinex and mtgox, and mtgox is the most popular exchange, then the incentives might drive all the votes to aggregate around the gox-btc/usd price specifically, and at that point it is entirely unclear what would happen when mtgox gets hacked and the price on that exchange alone, and not the others, falls to $100. everyone may well end up following their individual incentives and sticking to each other to the protocol's collective doom. how to deal with these situations and whether or not they are even significant is an entirely empirical issue; it is hard to say what the real world will do beforehand. formalizing the protocol, we have the following: every set of n blocks (here, we set n = 100) constitutes a separate "epoch". we define the epoch number as floor(block.number / 100), and we define the block number modulo 100 to be the "residual". if the residual is less than 50, then anyone can submit a transaction with any value v and hash h = sha3([msg.sender, r, p]), where p is their estimate of the price of 1 us cent in wei (remember, 1 wei = 10-18 ether, and 1 cent = 10-2 usd) and r is a random number. if the residual is greater than 50, then anyone who submitted a hash can submit p, and the contract will check if sha3([msg.sender, p]) matches the hash. at the end of the epoch (or, more precisely, at the point of the first "ping" during the next epoch), everyone who submitted a value for p between the 25th and 75th percentile, weighted by deposit, gets their deposit back plus a small reward, everyone else gets their deposit minus a small penalty, and the median value is taken to be the true uscent/wei price. everyone who failed to submit a valid value for p gets their deposit back minus a small penalty. note that there are possible optimizations to the protocol; for example, one might introduce a feature that allows anyone with a particular p value to steal the deposit from whoever submitted the hash, making it impractical to share one's p to try to influence people's votes before residual 50 hits and the second phase starts. however, to keep this example from getting too complicated we will not do this; additionally, i personally am skeptical of "forced private data revelation" strategies in general because i predict that many of them will become useless with the eventual advent of generalized zero-knowledge proofs, fully homomorphic encryption and obfuscation. for example, one might imagine an attacker beating such a scheme by supplying a zero-knowledge proof that their p value is within a particular 1015 wei-wide range, giving enough information to give users a target but not enough to practically locate the exact value of p . given these concerns, and given the desire for simplicity, for now the simple two-round protocol with no bells-and-whistles is best. before we start coding schellingcoin itself, there is one other contract that we will need to create: a sorting function. the only way to calculate the median of a list of numbers and determine who is in a particular percentile range is to sort the list, so we will want a generalized function to do that. for added utility, we will make our sorting function generic: we will sort pairs instead of integers. thus, for examples, [30, 1, 90, 2, 70, 3, 50, 4] would become [ 30, 1, 50, 4, 70, 3, 90, 2 ]. using this function, one can sort a list containing any kind of object simply by making an array of pairs where the first number is the key to sort by and the second number is a pointer to the object in parent memory or storage. here's the code: if msg.datasize == 0: return([], 0) else: low = array(msg.datasize) lsz = 0 high = array(msg.datasize) hsz = 0 i = 2 while i < msg.datasize: if msg.data[i] < msg.data[0]: low[lsz] = msg.data[i] low[lsz + 1] = msg.data[i + 1] lsz += 2 else: high[hsz] = msg.data[i] high[hsz + 1] = msg.data[i + 1] hsz += 2 i = i + 2 low = call(contract.address, low, lsz, lsz) high = call(contract.address, high, hsz, hsz) o = array(msg.datasize) i = 0 while i < lsz: o[i] = low[i] i += 1 o[lsz] = msg.data[0] o[lsz + 1] = msg.data[1] j = 0 while j < hsz: o[lsz + 2 + j] = high[j] j += 1 return(o, msg.datasize) computer students may recognize this as a quicksort implementation; the idea is that we first split the list into two, with one half containing everything less than the first item and the other half containing everything greater, then we recursively sort the first and second lists (the recursion terminates eventually, since eventually the sub-lists will have zero or one items, in which case we just return those values directly), and finally we concatenate output = sorted_less_than_list + first item + sorted_greater_than_list and return that array. now, putting that into "quicksort_pairs.se", let's build the code for the actual schellingcoin. feel free to go to the github to see the code all in one piece; here, we will go through it a few lines at a time. first, some initialization code: init: contract.storage[0] = block.number contract.storage[3] = create('quicksort_pairs.se') code: hashes = 2^160 values = 2^170 the first code block sets contract storage index 0 to the current block number at initialization time, and then creates a quicksort contract and saves that in storage index 3. note that theoretically you would want to just create the quicksort contract once and refer to it by address; we're just doing an inline create for simplicity and to show the feature. in the code we start off by declaring two variables to serve as pseudo-constants; hashes = 2160 as the pointer for where we store hashes, and values = 2170 as the pointer for where we store values from the second phase. now, from here let's skip to the bottom half of the code, because that turns out to be more convenient and it's the code that actually gets run "first" over the course of the contract's lifetime. # hash submission if msg.data[0] == 1: if block.number % 100 < 50: cur = contract.storage[1] pos = hashes + cur * 3 contract.storage[pos] = msg.data[1] contract.storage[pos + 1] = msg.value contract.storage[pos + 2] = msg.sender contract.storage[1] = cur + 1 return(cur) # value submission elif msg.data[0] == 2: if sha3([msg.sender, msg.data[3], msg.data[2]], 2) == contract.storage[hashes + msg.data[1] * 3]: contract.storage[values + msg.data[1]] = msg.data[2] return(1) # balance request elif msg.data[0] == 3: return(contract.balance) # value request else: return(contract.storage[2]) the first important paradigm that we see here is using msg.data[0] to refer to a "message type"; messages with zeroth data item 1 are hash submissions, 2 are value submissions, 3 are balance requests and 4 are requests for the current uscent/wei price. this is a standard interface that you will likely see across very many contracts. the first clause, the one for submitting hashes, is somewhat involved, so let us break it down step by step. the primary purpose here is to allow people to submit hashes, and record submissions in storage. to that end, the contract is storing the data sequentially in storage starting at index 2160. we need to store three pieces of data the actual hash, the size of the accompanying deposit, and the sender address, for each hash, so we do that. we also use storage index 1 to store how many hashes have already been submitted. thus, if two hashes have been submitted, storage will look something like this: the precise instructions in the clause are: proceed only if the residual is less than 50. set the variable cur to storage index 1, where we are going to be storing the number of hashes that have already been submitted set the variable pos to the index in storage in which we will be putting the new hash save the hash (supplied as the first data item), the sender address and the value in storage set the new number of hashes to cur + 1 return the index of the hash supplied technically, if the only users of schellingcoin are people, step 5 is unnecessary; although the index will be necessary in a later step, a smart client could potentially simply scan the cur variable immediately after the transaction, removing the need for the opcodes needed to handle the return. however, since we expect that in ethereum we will have plenty of instances of contracts using other contracts, we will provide the return value as a habit of good machine interface. the next clause is for submitting values. here, we ask for two data items as input: the index where the hash was saved during the first step of the protocol (this is the return value of the previous clause), and the actual value. we then hash the sender and value together, and if the hash matches then we save the result in another place in contract storage; an alternative approach is to use one single starting storage location and simply have four slots per hash instead of three. we return 1 is successful, and nothing for a failure. the third and fourth clauses are simply trivial data requests; the third is a balance check, and the fourth returns the contract's current view of the price. that's all for the interface side of the contract; however, the one part that we still need to do is the part that actually aggregates the votes. we'll break that up into parts. first, we have: hashes = 2^160 values = 2^170 if block.number / 100 > contract.storage[0] / 100: # sort all hashes n = contract.storage[1] o = array(n) i = 0 j = 0 while i < n: if contract.storage[values + i]: o[j] = contract.storage[values + i] o[j + 1] = i j += 2 i += 1 values = call(contract.storage[3], o, j, j) first, we use storage index 0 to store the last accessed epoch, and we check if the current epoch is higher than the last accessed epoch. if it is, then that signals the start of a new epoch, so we need to process all the votes and clear the contract for the next epoch. we start off by copying the values that have been submitted to an array (values that have not been submitted, ie. zeroes, are not put into this array). we keep two running counters, i and j; the counter i runs through all value slots, but the counter j counts only the value slots that have something inside them. note that the array that we produce is of the form [ val1, index1, val2, index2 ... ], where index1 etc are the indices of the associated values in the original values array in contract storage, thus for example, the following values would lead to the following array: then, we send that array through the quicksort contract, which sorts data pairs in the array. after the sort, we end up with: now, what we have is a sorted list of all the values that people have submitted, alongside pointers to where the associated metadata is stored in cold storage. the next part of the code will handle three things simultaneously. first, it will compute the total amount that has been deposited; this is useful in figuring out the median. second, we will make two arrays to represent deposits and their associated addresses, and we will remove that data from the contract. finally, we will 99.9% refund anyone who did not submit a value. theoretically, we could make it a 70% refund or a 0% refund, but that might make the contract too risky for people to throw their life savings in (which is actually what we want in a proof-of-stake-weighted system; the more ether is thrown in by legitimate users the harder it is for an attacker to muster enough funds to launch an attack). here's the code; feel free to understand each line yourself: # calculate total deposit, refund non-submitters and # cleanup deposits = array(j / 2) addresses = array(j / 2) i = 0 total_deposit = 0 while i < j / 2: base_index = hashes + values[i * 2 + 1] * 3 contract.storage[base_index] = 0 deposits[i] = contract.storage[base_index + 1] contract.storage[base_index + 1] = 0 addresses[i] = contract.storage[base_index + 2] contract.storage[base_index + 2] = 0 if contract.storage[values + values[i * 2 + 1]]: total_deposit += deposits[i] else: send(addresses[i], deposits[i] * 999 / 1000) i += 1 now, we come to the last part of the code, the part the computes the median and rewards people. according to the specification, we need to reward everyone between the 25th and 75th percentile, and take the median (ie. 50th percentile) as the truth. to actually do this, we needed to first sort the data; now that the data is sorted, however, it's as simple as maintaining a running counter of "total deposited value of everything in the list up to this point". if that value is between 25% and 75% of the total deposit, then we send a reward slightly greater than what they sent in, otherwise we send a slightly smaller reward. here is the code: inverse_profit_ratio = total_deposit / (contract.balance / 1000) + 1 # reward everyone i = 0 running_deposit_sum = 0 halfway_passed = 0 while i < j / 2: new_deposit_sum = running_deposit_sum + deposits[i] if new_deposit_sum > total_deposit / 4 and running_deposit_sum < total_deposit * 3 / 4: send(addresses[i], deposits[i] + deposits[i] / inverse_profit_ratio * 3) else: send(addresses[i], deposits[i] deposits[i] / inverse_profit_ratio) if not halfway_passed and new_deposit_sum > total_deposit / 2: contract.storage[2] = contract.storage[values + i] halfway_passed = 1 contract.storage[values + i] = 0 running_deposit_sum = new_deposit_sum i += 1 contract.storage[0] = block.number contract.storage[1] = 0 at the same time, you can see we also zero out the values in contract storage, and we update the epoch and reset the number of hashes to zero. the first value that we calculate, the "inverse profit ratio", is basically the inverse of the "interest rate" you get on your deposit; if inverse_profit_ratio = 33333, and you submitted 1000000 wei, then you get 1000090 wei back if you are close to the median and 999970 if you are not (ie. your expected return is 1000030 wei). note that although this amount is tiny, it happens per hundred blocks, so really it is quite large. and that's all there is to it. if you want to test, then try running the following python script: import pyethereum t = pyethereum.tester s = t.state() s.mine(123) c = s.contract('schellingcoin.se') c2 = s.contract('schellinghelper.se') vals = [[125, 200], [126, 900], [127, 500], [128, 300], [133, 300], [135, 150], [135, 150]] s.send(t.k9, c, 10**15) print "submitting hashes" for i, v in enumerate(vals): print s.send(t.keys[i], c, v[1], [1] + s.send(t.keys[i], c2, 0, [v[0], 12378971241241])) s.mine(50) print "submitting vals" for i, v in enumerate(vals): if i != 5: print s.send(t.keys[i], c, 0, [2, i, v[0], 12378971241241]) else: print s.send(t.keys[i], c, 0, [2, i, 4]) print "final check" s.mine(50) print s.send(t.k9, c, 0, [4]) before running the script, be sure to fill the 'schellinghelper.se' file with return(sha3([msg.sender, msg.data[0], msg.data[1]], 3)); here, we're just being lazy and using serpent itself to help us put the hash together; in reality, this should definitely be done off-chain. if you do that, and run the script, the last value printed by the contract should return 127. note that this contract as it stands is not really scalable by itself; at 1000+ users, whoever supplies the first transaction at the start of each epoch would need to pay a very large amount of gas. the way to fix this economically is of course to reward the submitter of the transaction, and take a flat fee off every participant to pay for the reward. also, however, the interest rate per epoch is tiny, so it may already not be worth it for users to participate unless they have a signigicant amount of cash, and the flat fee may make this problem even worse. to allow people to participate with small amounts of ether, the simplest solution is to create a "stake pool" where people put their ether into a contract for the long term, and then the pool votes together, randomly selecting a participant weighted by stake to supply the value to vote for in each epoch. this would reduce the load from two transactions per user per epoch to three transactions per pool per epoch (eg. 1 pool = 1000 users) plus one transaction per user to deposit/withdraw. note that, unlike bitcoin mining pools, this stake pool is completely decentralized and blockchain-based, so it introduces at most very small centralization risks. however, this is an instructive example to show how a single contract or dao may end up leading to an entire ecosystem of infrastructure working on the blockchain with contracts talking to each other; a specialized schellingcoin blockchain would not be able to invent pooling mechanisms after the fact and integrate them so efficiently. as far as applications go, the most immediate one is contracts for difference, and eventually a decentralized cryptographic us dollar; if you want to see an attempt at such a contract see here, although that code is almost certainly vulnerable to market manipulation attacks (buy a very large amount of usd inside the system, then buy usd on the market to move the price 0.5%, then sell the usd inside the system for a quick 0.3% profit). the core idea behind the decentralized crypto-dollar is simple: have a bank with two currencies, usd and ether (or rather, uscent and wei), with the ability to have a positive or negative quantity of dollars, and manipulate the interest rate on dollar deposits in order to keep the contract's net dollar exposure always close to zero so that the contract does not have any net obligations in currencies that it does not have the ability to hold. a simpler approach would simply be to have an expanding-supply currency that adjusts its supply function to target the usd, but that is problematic because there is no protection if the value falls too much. these kinds of applications, however, will likely take quite a long time (in crypto terms; quite fast in traditional finance terms of course) to get built. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum foundation open call re: board selection | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum foundation open call re: board selection posted by vitalik buterin on april 10, 2015 organizational with ethereum getting close to its first official launch with frontier, we have spent the last week of february at our holon in switzerland discussing what the future of the ethereum foundation is going to look like. since the start of the project, one of our primary dreams has been to not just deliver a world-class product, but also build a world-class organization, with quality of governance and transparency suitable for a foundation that would be tasked with helping to organize the maintenance of the ethereum code for potentially decades to come. to that end, we have taken a number of steps including an effort to hire new administrative staff and an internal branding shift, and now we are officially announcing our search for new members to join our board of directors. first, an introduction to roughly how the foundation is structured now from an organizational standpoint. stiftung ethereum (that's german for "ethereum foundation") is a non-profit organization registered in switzerland; specifically, in the wonderful town of zug: the stiftung is the body that is, ultimately, tasked with managing the funds and determining the long-term vision and direction of ethereum. the foundation is legally controlled by a body known as the foundation council; however, a larger advisory group also exists that has been involved in providing input for decision-making. there are a number of legal entities registered in multiple countries across europe, and these entities each employ the individuals housed in their respective countries. the centers in switzerland and berlin are the largest. the foundation council is intended to make decisions occasionally and focus on the long term; for day-to-day decisions regarding development an "executive" consisting of myself, gavin wood and jeffrey wilcke is in charge, with considerable assistance from aeron buchanan, jutta steiner, kelley becker, frithjof weinert and others on administrative and security management. we are very grateful for the continued service of the current foundation advisory board and council, and have had a successful first year under their leadership. however, now that the organization's scope and size had expanded considerably, we are looking to simultaneously (i) introduce new members to be part of the foundation council, and (ii) transition the foundation's activities and operations to more closely resemble a mature organization. board members would be required to, at the minimum, come to switzerland two to five times per year in order to attend official board meetings, where the executive team will provide an update on the current state of the project and we will discuss directions and if necessary pass resolutions relating to the foundation. anyone on the board is free to make further contributions beyond this. the new board is expected to be finalized and inaugurated at the end of june. the ideal characteristics that we are looking for in potential board candidates include: experience participating in non-profit boards status as a leader and innovator in a field (for-profit or non-profit) which could be benefited by blockchain and decentralized application technology, ideally outside of the existing "cryptocurrency" space. experience and interest in helping with long-term fundraising efforts minimal conflicts of interest (given the prior criterion, conflicts of interest are inevitable, but this does mean that we prefer individuals who are either involved primarily in non-profit activities or whose for-profit activities we can reasonably expect will not intersect much with their position at ethereum) excitement and interest in decentralized technology lack of prior unethical or illegal behavior, particularly in the areas of embezzlement, insider trading or fraud diverse set of industry backgrounds and national and cultural backgrounds (eg. hoping for a balance between north/south america, europe and asia) if you feel that you meet most or all of these requirements, or know someone who does, please do not hesitate to send an email to kelley@ethereum.org. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle the dawn of hybrid layer 2 protocols 2019 aug 28 see all posts special thanks to the plasma group team for review and feedback current approaches to layer 2 scaling basically, plasma and state channels are increasingly moving from theory to practice, but at the same time it is becoming easier to see the inherent challenges in treating these techniques as a fully fledged scaling solution for ethereum. ethereum was arguably successful in large part because of its very easy developer experience: you write a program, publish the program, and anyone can interact with it. designing a state channel or plasma application, on the other hand, relies on a lot of explicit reasoning about incentives and application-specific development complexity. state channels work well for specific use cases such as repeated payments between the same two parties and two-player games (as successfully implemented in celer), but more generalized usage is proving challenging. plasma, particularly plasma cash, can work well for payments, but generalization similarly incurs challenges: even implementing a decentralized exchange requires clients to store much more history data, and generalizing to ethereum-style smart contracts on plasma seems extremely difficult. but at the same time, there is a resurgence of a forgotten category of "semi-layer-2" protocols a category which promises less extreme gains in scaling, but with the benefit of much easier generalization and more favorable security models. a long-forgotten blog post from 2014 introduced the idea of "shadow chains", an architecture where block data is published on-chain, but blocks are not verified by default. rather, blocks are tentatively accepted, and only finalized after some period of time (eg. 2 weeks). during those 2 weeks, a tentatively accepted block can be challenged; only then is the block verified, and if the block proves to be invalid then the chain from that block on is reverted, and the original publisher's deposit is penalized. the contract does not keep track of the full state of the system; it only keeps track of the state root, and users themselves can calculate the state by processing the data submitted to the chain from start to head. a more recent proposal, zk rollup, does the same thing without challenge periods, by using zk-snarks to verify blocks' validity. anatomy of a zk rollup package that is published on-chain. hundreds of "internal transactions" that affect the state (ie. account balances) of the zk rollup system are compressed into a package that contains ~10 bytes per internal transaction that specifies the state transitions, plus a ~100-300 byte snark proving that the transitions are all valid. in both cases, the main chain is used to verify data availability, but does not (directly) verify block validity or perform any significant computation, unless challenges are made. this technique is thus not a jaw-droppingly huge scalability gain, because the on-chain data overhead eventually presents a bottleneck, but it is nevertheless a very significant one. data is cheaper than computation, and there are ways to compress transaction data very significantly, particularly because the great majority of data in a transaction is the signature and many signatures can be compressed into one through many forms of aggregation. zk rollup promises 500 tx/sec, a 30x gain over the ethereum chain itself, by compressing each transaction to a mere ~10 bytes; signatures do not need to be included because their validity is verified by the zero-knowledge proof. with bls aggregate signatures a similar throughput can be achieved in shadow chains (more recently called "optimistic rollup" to highlight its similarities to zk rollup). the upcoming istanbul hard fork will reduce the gas cost of data from 68 per byte to 16 per byte, increasing the throughput of these techniques by another 4x (that's over 2000 transactions per second). so what is the benefit of data on-chain techniques such as zk/optimistic rollup versus data off-chain techniques such as plasma? first of all, there is no need for semi-trusted operators. in zk rollup, because validity is verified by cryptographic proofs there is literally no way for a package submitter to be malicious (depending on the setup, a malicious submitter may cause the system to halt for a few seconds, but this is the most harm that can be done). in optimistic rollup, a malicious submitter can publish a bad block, but the next submitter will immediately challenge that block before publishing their own. in both zk and optimistic rollup, enough data is published on chain to allow anyone to compute the complete internal state, simply by processing all of the submitted deltas in order, and there is no "data withholding attack" that can take this property away. hence, becoming an operator can be fully permissionless; all that is needed is a security deposit (eg. 10 eth) for anti-spam purposes. second, optimistic rollup particularly is vastly easier to generalize; the state transition function in an optimistic rollup system can be literally anything that can be computed within the gas limit of a single block (including the merkle branches providing the parts of the state needed to verify the transition). zk rollup is theoretically generalizeable in the same way, though in practice making zk snarks over general-purpose computation (such as evm execution) is very difficult, at least for now. third, optimistic rollup is much easier to build clients for, as there is less need for second-layer networking infrastructure; more can be done by just scanning the blockchain. but where do these advantages come from? the answer lies in a highly technical issue known as the data availability problem (see note, video). basically, there are two ways to try to cheat in a layer-2 system. the first is to publish invalid data to the blockchain. the second is to not publish data at all (eg. in plasma, publishing the root hash of a new plasma block to the main chain but without revealing the contents of the block to anyone). published-but-invalid data is very easy to deal with, because once the data is published on-chain there are multiple ways to figure out unambiguously whether or not it's valid, and an invalid submission is unambiguously invalid so the submitter can be heavily penalized. unavailable data, on the other hand, is much harder to deal with, because even though unavailability can be detected if challenged, one cannot reliably determine whose fault the non-publication is, especially if data is withheld by default and revealed on-demand only when some verification mechanism tries to verify its availability. this is illustrated in the "fisherman's dilemma", which shows how a challenge-response game cannot distinguish between malicious submitters and malicious challengers: fisherman's dilemma. if you only start watching the given specific piece of data at time t3, you have no idea whether you are living in case 1 or case 2, and hence who is at fault. plasma and channels both work around the fisherman's dilemma by pushing the problem to users: if you as a user decide that another user you are interacting with (a counterparty in a state channel, an operator in a plasma chain) is not publishing data to you that they should be publishing, it's your responsibility to exit and move to a different counterparty/operator. the fact that you as a user have all of the previous data, and data about all of the transactions you signed, allows you to prove to the chain what assets you held inside the layer-2 protocol, and thus safely bring them out of the system. you prove the existence of a (previously agreed) operation that gave the asset to you, no one else can prove the existence of an operation approved by you that sent the asset to someone else, so you get the asset. the technique is very elegant. however, it relies on a key assumption: that every state object has a logical "owner", and the state of the object cannot be changed without the owner's consent. this works well for utxo-based payments (but not account-based payments, where you can edit someone else's balance upward without their consent; this is why account-based plasma is so hard), and it can even be made to work for a decentralized exchange, but this "ownership" property is far from universal. some applications, eg. uniswap don't have a natural owner, and even in those applications that do, there are often multiple people that can legitimately make edits to the object. and there is no way to allow arbitrary third parties to exit an asset without introducing the possibility of denial-of-service (dos) attacks, precisely because one cannot prove whether the publisher or submitter is at fault. there are other issues peculiar to plasma and channels individually. channels do not allow off-chain transactions to users that are not already part of the channel (argument: suppose there existed a way to send $1 to an arbitrary new user from inside a channel. then this technique could be used many times in parallel to send $1 to more users than there are funds in the system, already breaking its security guarantee). plasma requires users to store large amounts of history data, which gets even bigger when different assets can be intertwined (eg. when an asset is transferred conditional on transfer of another asset, as happens in a decentralized exchange with a single-stage order book mechanism). because data-on-chain computation-off-chain layer 2 techniques don't have data availability issues, they have none of these weaknesses. zk and optimistic rollup take great care to put enough data on chain to allow users to calculate the full state of the layer 2 system, ensuring that if any participant disappears a new one can trivially take their place. the only issue that they have is verifying computation without doing the computation on-chain, which is a much easier problem. and the scalability gains are significant: ~10 bytes per transaction in zk rollup, and a similar level of scalability can be achieved in optimistic rollup by using bls aggregation to aggregate signatures. this corresponds to a theoretical maximum of ~500 transactions per second today, and over 2000 post-istanbul. but what if you want more scalability? then there is a large middle ground between data-on-chain layer 2 and data-off-chain layer 2 protocols, with many hybrid approaches that give you some of the benefits of both. to give a simple example, the history storage blowup in a decentralized exchange implemented on plasma cash can be prevented by publishing a mapping of which orders are matched with which orders (that's less than 4 bytes per order) on chain: left: history data a plasma cash user needs to store if they own 1 coin. middle: history data a plasma cash user needs to store if they own 1 coin that was exchanged with another coin using an atomic swap. right: history data a plasma cash user needs to store if the order matching is published on chain. even outside of the decentralized exchange context, the amount of history that users need to store in plasma can be reduced by having the plasma chain periodically publish some per-user data on-chain. one could also imagine a platform which works like plasma in the case where some state does have a logical "owner" and works like zk or optimistic rollup in the case where it does not. plasma developers are already starting to work on these kinds of optimizations. there is thus a strong case to be made for developers of layer 2 scalability solutions to move to be more willing to publish per-user data on-chain at least some of the time: it greatly increases ease of development, generality and security and reduces per-user load (eg. no need for users storing history data). the efficiency losses of doing so are also overstated: even in a fully off-chain layer-2 architecture, users depositing, withdrawing and moving between different counterparties and providers is going to be an inevitable and frequent occurrence, and so there will be a significant amount of per-user on-chain data regardless. the hybrid route opens the door to a relatively fast deployment of fully generalized ethereum-style smart contracts inside a quasi-layer-2 architecture. see also: introducing the ovm blog post by karl floersch related ideas by john adler on silos | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on silos posted by vitalik buterin on december 31, 2014 research & development one of the criticisms that many people have made about the current direction of the cryptocurrency space is the increasing amount of fragmentation that we are seeing. what was earlier perhaps a more tightly bound community centered around developing the common infrastructure of bitcoin is now increasingly a collection of "silos", discrete projects all working on their own separate things. there are a number of developers and researchers who are either working for ethereum or working on ideas as volunteers and happen to spend lots of time interacting with the ethereum community, and this set of people has coalesced into a group dedicated to building out our particular vision. another quasi-decentralized collective, bitshares, has set their hearts on their own vision, combining their particular combination of dpos, market-pegged assets and vision of blockchain as decentralized autonomous corporation as a way of reaching their political goals of free-market libertarianism and a contract free society. blockstream, the company behind "sidechains", has likewise attracted their own group of people and their own set of visions and agendas and likewise for truthcoin, maidsafe, nxt, and many others. one argument, often raised by bitcoin maximalists and sidechains proponents, is that this fragmentation is harmful to the cryptocurrency ecosystem instead of all going our own separate ways and competing for users, we should all be working together and cooperating under bitcoin's common banner. as fabian brian crane summarizes: one recent event that has further inflamed the discussion is the publication of the sidechains proposal. the idea of sidechains is to allow the trustless innovation of altcoins while offering them the same monetary base, liquidity and mining power of the bitcoin network. for the proponents, this represents a crucial effort to rally the cryptocurrency ecosystem behind its most successful project and to build on the infrastructure and ecosystem already in place, instead of dispersing efforts in a hundred different directions. even to those who disagree with bitcoin maximalism, this seems like a rather reasonable point, and even if the cryptocurrency community should not all stand together under the banner of "bitcoin" one may argue that we need to all stand together somehow, working to build a more unified ecosystem. if bitcoin is not powerful enough to be a viable backbone for life, the crypto universe and everything, then why not build a better and more scalable decentralized computer instead and build everything on that? hypercubes certainly seem powerful enough to be worth being a maximalist over, if you're the sort of person to whom one-x-to-rule-them-all proposals are intuitively appealing, and the members of bitshares, blockstream and other "silos" are often quite eager to believe the same thing about their own particular solutions, whether they are based on merged-mining, dpos plus bitassets or whatever else. so why not? if there truly is one consensus mechanism that is best, why should we not have a large merger between the various projects, come up with the best kind of decentralized computer to push forward as a basis for the crypto-economy, and move forward together under one unified system? in some respects, this seems noble; "fragmentation" certainly has undesirable properties, and it is natural to see "working together" as a good thing. in reality, however, while more cooperation is certainly useful, and this blog post will later describe how and why, desires for extreme consolidation or winner-take-all are to a large degree exactly wrong not only is fragmentation not all that bad, but rather it's inevitable, and arguably the only way that this space can reasonably prosper. agree to disagree why has fragmentation been happening, and why should we continue to let it happen? to the first question, and also simultaneously to the second, the answer is simple: we fragment because we disagree. particularly, consider some of the following claims, all of which i believe in, but which are in many cases a substantial departure from the philosophies of many other people and projects: i do not think that weak subjectivity is all that much of a problem. however, much higher degrees of subjectivity and intrinsic reliance on extra-protocol social consensus i am still not comfortable with. i consider bitcoin's $600 million/year wasted electricity on proof of work to be an utter environmental and economic tragedy. i believe asics are a serious problem, and that as a result of them bitcoin has become qualitatively less secure over the past two years. i consider bitcoin (or any other fixed-supply currency) to be too incorrigibly volatile to ever be a stable unit of account, and believe that the best route to cryptocurrency price stability is by experimenting with intelligently designed flexible monetary policies (ie. not "the market" or "the bitcoin central bank"). however, i am not interested in bringing cryptocurrency monetary policy under any kind of centralized control. i have a substantially more anti-institutional/libertarian/anarchistic mindset than some people, but substantially less so than others (and am incidentally not an austrian economist). in general, i believe there is value to both sides of the fence, and believe strongly in being diplomatic and working together to make the world a better place. i am not in favor of there being one-currency-to-rule-them-all, in the crypto-economy or anywhere. i think token sales are an awesome tool for decentralized protocol monetization, and that everyone attacking the concept outright is doing a disservice to society by threatening to take away a beautiful thing. however, i do agree that the model as implemented by us and other groups so far has its flaws and we should be actively experimenting with different models that try to align incentives better i believe futarchy is promising enough to be worth trying, particularly in a blockchain governance context. i consider economics and game theory to be a key part of cryptoeconomic protocol analysis, and consider the primary academic deficit of the cryptocurrency community to be not ignorance of advanced computer science, but rather economics and philosophy. we should reach out to http://lesswrong.com/ more. i see one of the primary reasons why people will adopt decentralized technologies (blockchains, whisper, dhts) in practice to be the simple fact that software developers are lazy, and do not wish to deal with the complexities of maintaining a centralized website. i consider the blockchain-as-decentralized-autonomous-corporation metaphor to be useful, but limited. particularly, i believe that we as cryptocurrency developers should be taking advantage of this perhaps brief period in which cryptocurrency is still an idealist-controlled industry to design institutions that maximize utilitarian social welfare metrics, not profit (no, they are not equivalent, primarily because of these). there are probably very few people who agree with me on every single one of the items above. and it is not just myself that has my own peculiar opinions. as another example, consider the fact that the cto of opentransactions, chris odom, says things like this: what is needed is to replace trusted entities with systems of cryptographic proof. any entity that you see in the bitcoin community that you have to trust is going to go away, it's going to cease to exist ... satoshi's dream was to eliminate [trusted] entities entirely, either eliminate the risk entirely or distribute the risk in a way that it's practically eliminated. meanwile, certain others feel the need to say things like this: put differently, commercially viable reduced-trust networks do not need to protect the world from platform operators. they will need to protect platform operators from the world for the benefit of the platform’s users. of course, if you see the primary benefit of cryptocurrency as being regulation avoidance then that second quote also makes sense, but in a way completely different from the way its original author intended but that once again only serves to show just how differently people think. some people see cryptocurrency as a capitalist revolution, others see it as an egalitarian revolution, and others see everything in between. some see human consensus as a very fragile and corruptible thing and cryptocurrency as a beacon of light that can replace it with hard math; others see cryptocurrency consensus as being only an extension of human consensus, made more efficient with technology. some consider the best way to achieve cryptoassets with dollar parity to be dual-coin financial derivative schemes; others see the simpler approach as being to use blockchains to represent claims on real-world assets instead (and still others think that bitcoin will eventually be more stable than the dollar all on its own). some think that scalability is best done by "scaling up"; others believe the ultimately superior option is "scaling out". of course, many of these issues are inherently political, and some involve public goods; in those cases, live and let live is not always a viable solution. if a particular platform enables negative externalities, or threatens to push society into a suboptimal equilibrium, then you cannot "opt out" simply by using your platform instead. there, some kind of network-effect-driven or even in extreme cases 51%-attack-driven censure may be necessary. in some cases, the differences are related to private goods, and are primarily simply a matter of empirical beliefs. if i believe that schellingdollar is the best scheme for price stability, and others prefer seignorage shares or nubits then after a few years or decades one model will prove to work better, replace its competition, and that will be that. in other cases, however, the differences will be resolved in a different way: it will turn out that the properties of some systems are better suited for some applications, and other systems better suited for other applications, and everything will naturally specialize into those use cases where it works best. as a number of commentators have pointed out, for decentralized consensus applications in the mainstream financial world, banks will likely not be willing to accept a network managed by anonymous nodes; in this case, something like ripple will be more useful. but for silk road 4.0, the exact opposite approach is the only way to go and for everything in between it's a cost-benefit analysis all the way. if users want networks specialized to performing specific functions highly efficiently, then networks will exist for that, and if users want a general purpose network with a high network effect between on-chain applications then that will exist as well. as david johnston points out, blockchains are like programming languages: they each have their own particular properties, and few developers religiously adhere to one language exclusively rather, we use each one in the specific cases for which it is best suited. room for cooperation however, as was mentioned earlier, this does not mean that we should simply go our own way and try to ignore or worse, actively sabotage, each other. even if all of our projects are necessarily specializing toward different goals, there is nevertheless a substantial opportunity for much less duplication of effort, and more cooperation. this is true on multiple levels. first, let us look at a model of the cryptocurrency ecosystem or, perhaps, a vision of what it might look like in 1-5 years time: ethereum has its own presence on pretty much every level: consensus: ethereum blockchain, data-availablility schelling-vote (maybe for ethereum 2.0) economics: ether, an independent token, as well as research into stablecoin proposals blockchain services: name registry off-chain services: whisper (messaging), web of trust (in progress) interop: btc-to-ether bridge (in progress) browsers: mist now, consider a few other projects that are trying to build holistic ecosystems of some kind. bitshares has at the least: consensus: dpos economics: btsx and bitassets blockchain services: bts decentralized exchange browsers: bitshares client (though not quite a browser in the same concept) maidsafe has: consensus: safe network economics: safecoin off-chain services: distributed hash table, maidsafe drive bittorrent has announced their plans for maelstrom, a project intended to serve a rather similar function to mist, albeit showcasing their own (not blockchain-based) technology. cryptocurrency projects generally all build a blockchain, a currency and a client of their own, although forking a single client is common for the less innovative cases. name registration and identity management systems are now a dime a dozen. and, of course, just about every project realizes that it has a need for some kind of reputation and web of trust. now, let us paint a picture of an alternative world. instead of having a collection of cleanly disjoint vertically integrated ecosystems, with each one building its own components for everything, imagine a world where mist could be used to access ethereum, bitshares, maidsafe or any other major decentralized infrastructure network, with new decentralized networks being installable much like the plugins for flash and java inside of chrome and firefox. imagine that the reputation data in the web of trust for ethereum could be reused in other projects as well. imagine storj running inside of maelstrom as a dapp, using maidsafe for a file storage backend, and using the ethereum blockchain to maintain the contracts that incentivize continued storage and downloading. imagine identities being automatically transferrable across any crypto-networks, as long as they use the same underlying cryptographic algorithms (eg. ecdsa + sha3). the key insight here is this: although some of the layers in the ecosystem are inextricably linked for example, a single dapp will often correspond to a single specific service on the ethereum blockchain in many cases the layers can easily be designed to be much more modular, allowing each product on each layer to compete separately on its own merits. browsers are perhaps the most separable component; most reasonably holistic lower level blockchain service sets have similar needs in terms of what applications can run on them, and so it makes sense for each browser to support each platform. off-chain services are also a target for abstraction; any decentralized application, regardless of what blockchain technology it uses, should be free to use whisper, swarm, ipfs or any other service that developers come up with. on-chain services, like data provision, can theoretically be built so as to interact with multiple chains. additionally, there are plenty of opportunities to collaborate on fundamental research and development. discussion on proof of work, proof of stake, stable currency systems and scalability, as well as other hard problems of cryptoeconomics can easily be substantially more open, so that the various projects can benefit from and be more aware of each other's developments. basic algorithms and best practices related to networking layers, cryptographic algorithm implementations and other low-level components can, and should, be shared. interoperability technologies should be developed to facilitate easy exchange and interaction between services and decentralized entities on one platform and another. the cryptocurrency research group is one initiative that we plan to initially support, with the hope that it will grow to flourish independently of ourselves, with the goal of promoting this kind of cooperation. other formal and informal institutions can doubtlessly help support the process. hopefully, in the future we will see many more projects existing in a much more modular fashion, living on only one or two layers of the cryptocurrency ecosystem and providing a common interface allowing any mechanism on any other layer to work with them. if the cryptocurrency space goes far enough, then even firefox and chrome may end up adapting themselves to process decentralized application protocols as well. a journey toward such an ecosystem is not something that needs to be rushed immediately; at this point, we have quite little idea of what kinds of blockchain-driven services people will be using in the first place, making it hard to determine exactly what kind of interoperability would actually be useful. however, things slowly but surely are taking their first few steps in that direction; eris's decerver, their own "browser" into the decentralized world, supports access to bitcoin, ethereum, their own thelonious blockchains as well as an ipfs content hosting network. there is room for many projects that are currently in the crypto 2.0 space to succeed, and so having a winner-take-all mentality at this point is completely unnecessary and harmful. all that we need to do right now to set off the journey on a better road is to live with the assumption that we are all building our own platforms, tuned to our own particular set of preferences and parameters, but at the end of the day a plurality of networks will succeed and we will need to live with that reality, so might as well start preparing for it now. happy new year, and looking forward to an exciting 2015 007 anno satoshii. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle on free speech 2019 apr 16 see all posts "a statement may be both true and dangerous. the previous sentence is such a statement." david friedman freedom of speech is a topic that many internet communities have struggled with over the last two decades. cryptocurrency and blockchain communities, a major part of their raison d'etre being censorship resistance, are especially poised to value free speech very highly, and yet, over the last few years, the extremely rapid growth of these communities and the very high financial and social stakes involved have repeatedly tested the application and the limits of the concept. in this post, i aim to disentangle some of the contradictions, and make a case what the norm of "free speech" really stands for. "free speech laws" vs "free speech" a common, and in my own view frustrating, argument that i often hear is that "freedom of speech" is exclusively a legal restriction on what governments can act against, and has nothing to say regarding the actions of private entities such as corporations, privately-owned platforms, internet forums and conferences. one of the larger examples of "private censorship" in cryptocurrency communities was the decision of theymos, the moderator of the /r/bitcoin subreddit, to start heavily moderating the subreddit, forbidding arguments in favor of increasing the bitcoin blockchain's transaction capacity via a hard fork. here is a timeline of the censorship as catalogued by john blocke: https://medium.com/johnblocke/a-brief-and-incomplete-history-of-censorship-in-r-bitcoin-c85a290fe43 here is theymos's post defending his policies: https://www.reddit.com/r/bitcoin/comments/3h9cq4/its_time_for_a_break_about_the_recent_mess/, including the now infamous line "if 90% of /r/bitcoin users find these policies to be intolerable, then i want these 90% of /r/bitcoin users to leave". a common strategy used by defenders of theymos's censorship was to say that heavy-handed moderation is okay because /r/bitcoin is "a private forum" owned by theymos, and so he has the right to do whatever he wants in it; those who dislike it should move to other forums: and it's true that theymos has not broken any laws by moderating his forum in this way. but to most people, it's clear that there is still some kind of free speech violation going on. so what gives? first of all, it's crucially important to recognize that freedom of speech is not just a law in some countries. it's also a social principle. and the underlying goal of the social principle is the same as the underlying goal of the law: to foster an environment where the ideas that win are ideas that are good, rather than just ideas that happen to be favored by people in a position of power. and governmental power is not the only kind of power that we need to protect from; there is also a corporation's power to fire someone, an internet forum moderator's power to delete almost every post in a discussion thread, and many other kinds of power hard and soft. so what is the underlying social principle here? quoting eliezer yudkowsky: there are a very few injunctions in the human art of rationality that have no ifs, ands, buts, or escape clauses. this is one of them. bad argument gets counterargument. does not get bullet. never. never ever never for ever. slatestarcodex elaborates: what does "bullet" mean in the quote above? are other projectiles covered? arrows? boulders launched from catapults? what about melee weapons like swords or maces? where exactly do we draw the line for "inappropriate responses to an argument"? a good response to an argument is one that addresses an idea; a bad argument is one that silences it. if you try to address an idea, your success depends on how good the idea is; if you try to silence it, your success depends on how powerful you are and how many pitchforks and torches you can provide on short notice. shooting bullets is a good way to silence an idea without addressing it. so is firing stones from catapults, or slicing people open with swords, or gathering a pitchfork-wielding mob. but trying to get someone fired for holding an idea is also a way of silencing an idea without addressing it. that said, sometimes there is a rationale for "safe spaces" where people who, for whatever reason, just don't want to deal with arguments of a particular type, can congregate and where those arguments actually do get silenced. perhaps the most innocuous of all is spaces like ethresear.ch where posts get silenced just for being "off topic" to keep the discussion focused. but there's also a dark side to the concept of "safe spaces"; as ken white writes: this may come as a surprise, but i'm a supporter of ‘safe spaces.' i support safe spaces because i support freedom of association. safe spaces, if designed in a principled way, are just an application of that freedom... but not everyone imagines "safe spaces" like that. some use the concept of "safe spaces" as a sword, wielded to annex public spaces and demand that people within those spaces conform to their private norms. that's not freedom of association aha. so making your own safe space off in a corner is totally fine, but there is also this concept of a "public space", and trying to turn a public space into a safe space for one particular special interest is wrong. so what is a "public space"? it's definitely clear that a public space is not just "a space owned and/or run by a government"; the concept of privately owned public spaces is a well-established one. this is true even informally: it's a common moral intuition, for example, that it's less bad for a private individual to commit violations such as discriminating against races and genders than it is for, say, a shopping mall to do the same. in the case or the /r/bitcoin subreddit, one can make the case, regardless of who technically owns the top moderator position in the subreddit, that the subreddit very much is a public space. a few arguments particularly stand out: it occupies "prime real estate", specifically the word "bitcoin", which makes people consider it to be the default place to discuss bitcoin. the value of the space was created not just by theymos, but by thousands of people who arrived on the subreddit to discuss bitcoin with an implicit expectation that it is, and will continue, to be a public space for discussing bitcoin. theymos's shift in policy was a surprise to many people, and it was not foreseeable ahead of time that it would take place. if, instead, theymos had created a subreddit called /r/bitcoinsmallblockers, and explicitly said that it was a curated space for small block proponents and attempting to instigate controversial hard forks was not welcome, then it seems likely that very few people would have seen anything wrong about this. they would have opposed his ideology, but few (at least in blockchain communities) would try to claim that it's improper for people with ideologies opposed to their own to have spaces for internal discussion. but back in reality, theymos tried to "annex a public space and demand that people within the space confirm to his private norms", and so we have the bitcoin community block size schism, a highly acrimonious fork and chain split, and now a cold peace between bitcoin and bitcoin cash. deplatforming about a year ago at deconomy i publicly shouted down craig wright, a scammer claiming to be satoshi nakamoto, finishing my explanation of why the things he says make no sense with the question "why is this fraud allowed to speak at this conference?" of course, craig wright's partisans replied back with.... accusations of censorship: did i try to "silence" craig wright? i would argue, no. one could argue that this is because "deconomy is not a public space", but i think the much better argument is that a conference is fundamentally different from an internet forum. an internet forum can actually try to be a fully neutral medium for discussion where anything goes; a conference, on the other hand, is by its very nature a highly curated list of presentations, allocating a limited number of speaking slots and actively channeling a large amount of attention to those lucky enough to get a chance to speak. a conference is an editorial act by the organizers, saying "here are some ideas and views that we think people really should be exposed to and hear". every conference "censors" almost every viewpoint because there's not enough space to give them all a chance to speak, and this is inherent to the format; so raising an objection to a conference's judgement in making its selections is absolutely a legitimate act. this extends to other kinds of selective platforms. online platforms such as facebook, twitter and youtube already engage in active selection through algorithms that influence what people are more likely to be recommended. typically, they do this for selfish reasons, setting up their algorithms to maximize "engagement" with their platform, often with unintended byproducts like promoting flat earth conspiracy theories. so given that these platforms are already engaging in (automated) selective presentation, it seems eminently reasonable to criticize them for not directing these same levers toward more pro-social objectives, or at the least pro-social objectives that all major reasonable political tribes agree on (eg. quality intellectual discourse). additionally, the "censorship" doesn't seriously block anyone's ability to learn craig wright's side of the story; you can just go visit their website, here you go: https://coingeek.com/. if someone is already operating a platform that makes editorial decisions, asking them to make such decisions with the same magnitude but with more pro-social criteria seems like a very reasonable thing to do. a more recent example of this principle at work is the #delistbsv campaign, where some cryptocurrency exchanges, most famously binance, removed support for trading bsv (the bitcoin fork promoted by craig weight). once again, many people, even reasonable people, accused this campaign of being an exercise in censorship, raising parallels to credit card companies blocking wikileaks: i personally have been a critic of the power wielded by centralized exchanges. should i oppose #delistbsv on free speech grounds? i would argue no, it's ok to support it, but this is definitely a much closer call. many #delistbsv participants like kraken are definitely not "anything-goes" platforms; they already make many editorial decisions about which currencies they accept and refuse. kraken only accepts about a dozen currencies, so they are passively "censoring" almost everyone. shapeshift supports more currencies but it does not support spank, or even knc. so in these two cases, delisting bsv is more like reallocation of a scarce resource (attention/legitimacy) than it is censorship. binance is a bit different; it does accept a very large array of cryptocurrencies, adopting a philosophy much closer to anything-goes, and it does have a unique position as market leader with a lot of liquidity. that said, one can argue two things in binance's favor. first of all, censorship is retaliating against a truly malicious exercise of censorship on the part of core bsv community members when they threatened critics like peter mccormack with legal letters (see peter's response); in "anarchic" environments with large disagreements on what the norms are, "an eye for an eye" in-kind retaliation is one of the better social norms to have because it ensures that people only face punishments that they in some sense have through their own actions demonstrated they believe are legitimate. furthermore, the delistings won't make it that hard for people to buy or sell bsv; coinex has said that they will not delist (and i would actually oppose second-tier "anything-goes" exchanges delisting). but the delistings do send a strong message of social condemnation of bsv, which is useful and needed. so there's a case to support all delistings so far, though on reflection binance refusing to delist "because freedom" would have also been not as unreasonable as it seems at first glance. it's in general absolutely potentially reasonable to oppose the existence of a concentration of power, but support that concentration of power being used for purposes that you consider prosocial as long as that concentration exists; see bryan caplan's exposition on reconciling supporting open borders and also supporting anti-ebola restrictions for an example in a different field. opposing concentrations of power only requires that one believe those concentrations of power to be on balance harmful and abusive; it does not mean that one must oppose all things that those concentrations of power do. if someone manages to make a completely permissionless cross-chain decentralized exchange that facilitates trade between any asset and any other asset, then being "listed" on the exchange would not send a social signal, because everyone is listed; and i would support such an exchange existing even if it supports trading bsv. the thing that i do support is bsv being removed from already exclusive positions that confer higher tiers of legitimacy than simple existence. so to conclude: censorship in public spaces bad, even if the public spaces are non-governmental; censorship in genuinely private spaces (especially spaces that are not "defaults" for a broader community) can be okay; ostracizing projects with the goal and effect of denying access to them, bad; ostracizing projects with the goal and effect of denying them scarce legitimacy can be okay. dark mode toggle a guide to 99% fault tolerant consensus 2018 aug 07 see all posts special thanks to emin gun sirer for review we've heard for a long time that it's possible to achieve consensus with 50% fault tolerance in a synchronous network where messages broadcasted by any honest node are guaranteed to be received by all other honest nodes within some known time period (if an attacker has more than 50%, they can perform a "51% attack", and there's an analogue of this for any algorithm of this type). we've also heard for a long time that if you want to relax the synchrony assumption, and have an algorithm that's "safe under asynchrony", the maximum achievable fault tolerance drops to 33% (pbft, casper ffg, etc all fall into this category). but did you know that if you add even more assumptions (specifically, you require observers, ie. users that are not actively participating in the consensus but care about its output, to also be actively watching the consensus, and not just downloading its output after the fact), you can increase fault tolerance all the way to 99%? this has in fact been known for a long time; leslie lamport's famous 1982 paper "the byzantine generals problem" (link here) contains a description of the algorithm. the following will be my attempt to describe and reformulate the algorithm in a simplified form. suppose that there are \(n\) consensus-participating nodes, and everyone agrees who these nodes are ahead of time (depending on context, they could have been selected by a trusted party or, if stronger decentralization is desired, by some proof of work or proof of stake scheme). we label these nodes \(0 ...n-1\). suppose also that there is a known bound \(d\) on network latency plus clock disparity (eg. \(d\) = 8 seconds). each node has the ability to publish a value at time \(t\) (a malicious node can of course propose values earlier or later than \(t\)). all nodes wait \((n-1) \cdot d\) seconds, running the following process. define \(x : i\) as "the value \(x\) signed by node \(i\)", \(x : i : j\) as "the value \(x\) signed by \(i\), and that value and signature together signed by \(j\)", etc. the proposals published in the first stage will be of the form \(v: i\) for some \(v\) and \(i\), containing the signature of the node that proposed it. if a validator \(i\) receives some message \(v : i[1] : ... : i[k]\), where \(i[1] ... i[k]\) is a list of indices that have (sequentially) signed the message already (just \(v\) by itself would count as \(k=0\), and \(v:i\) as \(k=1\)), then the validator checks that (i) the time is less than \(t + k \cdot d\), and (ii) they have not yet seen a valid message containing \(v\); if both checks pass, they publish \(v : i[1] : ... : i[k] : i\). at time \(t + (n-1) \cdot d\), nodes stop listening. at this point, there is a guarantee that honest nodes have all "validly seen" the same set of values. node 1 (red) is malicious, and nodes 0 and 2 (grey) are honest. at the start, the two honest nodes make their proposals \(y\) and \(x\), and the attacker proposes both \(w\) and \(z\) late. \(w\) reaches node 0 on time but not node 2, and \(z\) reaches neither node on time. at time \(t + d\), nodes 0 and 2 rebroadcast all values they've seen that they have not yet broadcasted, but add their signatures on (\(x\) and \(w\) for node 0, \(y\) for node 2). both honest nodes saw \({x, y, w}\). if the problem demands choosing one value, they can use some "choice" function to pick a single value out of the values they have seen (eg. they take the one with the lowest hash). the nodes can then agree on this value. now, let's explore why this works. what we need to prove is that if one honest node has seen a particular value (validly), then every other honest node has also seen that value (and if we prove this, then we know that all honest nodes have seen the same set of values, and so if all honest nodes are running the same choice function, they will choose the same value). suppose that any honest node receives a message \(v : i[1] : ... : i[k]\) that they perceive to be valid (ie. it arrives before time \(t + k \cdot d\)). suppose \(x\) is the index of a single other honest node. either \(x\) is part of \({i[1] ... i[k]}\) or it is not. in the first case (say \(x = i[j]\) for this message), we know that the honest node \(x\) had already broadcasted that message, and they did so in response to a message with \(j-1\) signatures that they received before time \(t + (j-1) \cdot d\), so they broadcast their message at that time, and so the message must have been received by all honest nodes before time \(t + j \cdot d\). in the second case, since the honest node sees the message before time \(t + k \cdot d\), then they will broadcast the message with their signature and guarantee that everyone, including \(x\), will see it before time \(t + (k+1) \cdot d\). notice that the algorithm uses the act of adding one's own signature as a kind of "bump" on the timeout of a message, and it's this ability that guarantees that if one honest node saw a message on time, they can ensure that everyone else sees the message on time as well, as the definition of "on time" increments by more than network latency with every added signature. in the case where one node is honest, can we guarantee that passive observers (ie. non-consensus-participating nodes that care about knowing the outcome) can also see the outcome, even if we require them to be watching the process the whole time? with the scheme as written, there's a problem. suppose that a commander and some subset of \(k\) (malicious) validators produce a message \(v : i[1] : .... : i[k]\), and broadcast it directly to some "victims" just before time \(t + k \cdot d\). the victims see the message as being "on time", but when they rebroadcast it, it only reaches all honest consensus-participating nodes after \(t + k \cdot d\), and so all honest consensus-participating nodes reject it. but we can plug this hole. we require \(d\) to be a bound on two times network latency plus clock disparity. we then put a different timeout on observers: an observer accepts \(v : i[1] : .... : i[k]\) before time \(t + (k 0.5) \cdot d\). now, suppose an observer sees a message an accepts it. they will be able to broadcast it to an honest node before time \(t + k \cdot d\), and the honest node will issue the message with their signature attached, which will reach all other observers before time \(t + (k + 0.5) \cdot d\), the timeout for messages with \(k+1\) signatures. retrofitting onto other consensus algorithms the above could theoretically be used as a standalone consensus algorithm, and could even be used to run a proof-of-stake blockchain. the validator set of round \(n+1\) of the consensus could itself be decided during round \(n\) of the consensus (eg. each round of a consensus could also accept "deposit" and "withdraw" transactions, which if accepted and correctly signed would add or remove validators into the next round). the main additional ingredient that would need to be added is a mechanism for deciding who is allowed to propose blocks (eg. each round could have one designated proposer). it could also be modified to be usable as a proof-of-work blockchain, by allowing consensus-participating nodes to "declare themselves" in real time by publishing a proof of work solution on top of their public key at th same time as signing a message with it. however, the synchrony assumption is very strong, and so we would like to be able to work without it in the case where we don't need more than 33% or 50% fault tolerance. there is a way to accomplish this. suppose that we have some other consensus algorithm (eg. pbft, casper ffg, chain-based pos) whose output can be seen by occasionally-online observers (we'll call this the threshold-dependent consensus algorithm, as opposed to the algorithm above, which we'll call the latency-dependent consensus algorithm). suppose that the threshold-dependent consensus algorithm runs continuously, in a mode where it is constantly "finalizing" new blocks onto a chain (ie. each finalized value points to some previous finalized value as a "parent"; if there's a sequence of pointers \(a \rightarrow ... \rightarrow b\), we'll call \(a\) a descendant of \(b\)). we can retrofit the latency-dependent algorithm onto this structure, giving always-online observers access to a kind of "strong finality" on checkpoints, with fault tolerance ~95% (you can push this arbitrarily close to 100% by adding more validators and requiring the process to take longer). every time the time reaches some multiple of 4096 seconds, we run the latency-dependent algorithm, choosing 512 random nodes to participate in the algorithm. a valid proposal is any valid chain of values that were finalized by the threshold-dependent algorithm. if a node sees some finalized value before time \(t + k \cdot d\) (\(d\) = 8 seconds) with \(k\) signatures, it accepts the chain into its set of known chains and rebroadcasts it with its own signature added; observers use a threshold of \(t + (k 0.5) \cdot d\) as before. the "choice" function used at the end is simple: finalized values that are not descendants of what was already agreed to be a finalized value in the previous round are ignored finalized values that are invalid are ignored to choose between two valid finalized values, pick the one with the lower hash if 5% of validators are honest, there is only a roughly 1 in 1 trillion chance that none of the 512 randomly selected nodes will be honest, and so as long as the network latency plus clock disparity is less than \(\frac{d}{2}\) the above algorithm will work, correctly coordinating nodes on some single finalized value, even if multiple conflicting finalized values are presented because the fault tolerance of the threshold-dependent algorithm is broken. if the fault tolerance of the threshold-dependent consensus algorithm is met (usually 50% or 67% honest), then the threshold-dependent consensus algorithm will either not finalize any new checkpoints, or it will finalize new checkpoints that are compatible with each other (eg. a series of checkpoints where each points to the previous as a parent), so even if network latency exceeds \(\frac{d}{2}\) (or even \(d\)), and as a result nodes participating in the latency-dependent algorithm disagree on which value they accept, the values they accept are still guaranteed to be part of the same chain and so there is no actual disagreement. once latency recovers back to normal in some future round, the latency-dependent consensus will get back "in sync". if the assumptions of both the threshold-dependent and latency-dependent consensus algorithms are broken at the same time (or in consecutive rounds), then the algorithm can break down. for example, suppose in one round, the threshold-dependent consensus finalizes \(z \rightarrow y \rightarrow x\) and the latency-dependent consensus disagrees between \(y\) and \(x\), and in the next round the threshold-dependent consensus finalizes a descendant \(w\) of \(x\) which is not a descendant of \(y\); in the latency-dependent consensus, the nodes who agreed \(y\) will not accept \(w\), but the nodes that agreed \(x\) will. however, this is unavoidable; the impossibility of safe-under-asynchrony consensus with more than \(\frac{1}{3}\) fault tolerance is a well known result in byzantine fault tolerance theory, as is the impossibility of more than \(\frac{1}{2}\) fault tolerance even allowing synchrony assumptions but assuming offline observers. secret sharing and erasure coding: a guide for the aspiring dropbox decentralizer | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search secret sharing and erasure coding: a guide for the aspiring dropbox decentralizer posted by vitalik buterin on august 16, 2014 research & development one of the more exciting applications of decentralized computing that have aroused a considerable amount of interest in the past year is the concept of an incentivized decentralized online file storage system. currently, if you want your files or data securely backed up "in the cloud", you have three choices (1) upload them to your own servers, (2) use a centralized service like google drive or dropbox or (3) use an existing decentralized file system like freenet. these approaches all have their own faults; the first has a high setup and maintenance cost, the second relies on a single trusted party and often involves heavy price markups, and the third is slow and very limited in the amount of space that it allows each user because it relies on users to volunteer storage. incentivized file storage protocols have the potential to provide a fourth way, providing a much higher quantity of storage and quality of service by incentivizing actors to participate without introducing centralization. a number of platforms, including storj, maidsafe, to some extent permacoin, and filecoin, are attempting to tackle this problem, and the problem seems simple in the sense that all the tools are either already there or en route to being built, and all we need is the implementation. however, there is one part of the problem that is particularly important: how do we properly introduce redundancy? redundancy is crucial to security; especially in a decentralized network that will be highly populated by amateur and casual users, we absolutely cannot rely on any single node to stay online. we could simply replicate the data, having a few nodes each store a separate copy, but the question is: can we do better? as it turns out, we absolutely can. merkle trees and challenge-response protocols before we get into the nitty gritty of redundancy, we will first cover the easier part: how do we create at least a basic system that will incentivize at least one party to hold onto a file? without incentivization, the problem is easy; you simply upload the file, wait for other users to download it, and then when you need it again you can make a request querying for the file by hash. if we want to introduce incentivization, the problem becomes somewhat harder but, in the grand scheme of things, still not too hard. in the context of file storage, there are two kinds of activities that you can incentivize. the first is the actual act of sending the file over to you when you request it. this is easy to do; the best strategy is a simple tit-for-tat game where the sender sends over 32 kilobytes, you send over 0.0001 coins, the sender sends over another 32 kilobytes, etc. note that for very large files without redundancy this strategy is vulnerable to extortion attacks quite often, 99.99% of a file is useless to you without the last 0.01%, so the storer has the opportunity to extort you by asking for a very high payout for the last block. the cleverest fix to this problem is actually to make the file itself redundant, using a special kind of encoding to expand the file by, say, 11.11% so that any 90% of this extended file can be used to recover the original, and then hiding the exact redundancy percentage from the storer; however, as it turns out we will discuss an algorithm very similar to this for a different purpose later, so for now, simply accept that this problem has been solved. the second act that we can incentivize is the act of holding onto the file and storing it for the long term. this problem is somewhat harder how can you prove that you are storing a file without actually transferring the whole thing? fortunately, there is a solution that is not too difficult to implement, using what has now hopefully established a familiar reputation as the cryptoeconomist's best friend: merkle trees. well, patricia merkle might be better in some cases, to be precise. athough here the plain old original merkle will do. the basic approach is this. first, split the file up into very small chunks, perhaps somewhere between 32 and 1024 bytes each, and add chunks of zeroes until the number of chunks reaches n = 2^k for some k (the padding step is avoidable, but it makes the algorithm simpler to code and explain). then, we build the tree. rename the n chunks that we received chunk[n] to chunk[2n-1] , and then rebuild chunks 1 to n-1 with the following rule: chunk[i] = sha3([chunk[2*i], chunk[2*i+1]]) . this lets you calculate chunks n/2 to n-1 , then n/4 to n/2 1 , and so forth going up the tree until there is one "root", chunk[1] . now, note that if you store only the root, and forget about chunk[2] ... chunk[2n-1], the entity storing those other chunks can prove to you that they have any particular chunk with only a few hundred bytes of data. the algorithm is relatively simple. first, we define a function partner(n) which gives n-1 if n is odd, otherwise n+1 in short, given a chunk find the chunk that it is hashed together with in order to produce the parent chunk. then, if you want to prove ownership of chunk[k] with n <= k <= 2n-1 (ie. any part of the original file), submit chunk[partner(k)], chunk[partner(k/2)] (division here is assumed to round down, so eg. 11 / 2 = 5), chunk[partner(k/4)] and so on down to chunk[1], alongside the actual chunk[k]. essentially, we're providing the entire "branch" of the tree going up from that node all the way to the root. the verifier will then take chunk[k] and chunk[partner(k)] and use that to rebuild chunk[k/2], use that and chunk[partner(k/2)] to rebuild chunk[k/4] and so forth until the verifier gets to chunk[1], the root of the tree. if the root matches, then the proof is fine; otherwise it's not. the proof of chunk 10 includes (1) chunk 10, and (2) chunks 11 (11 = partner(10) ), 4 (4 = partner(10/2) ) and 3 (3 = partner(10/4) ). the verification process involves starting off with chunk 10, using each partner chunk in turn to recompute first chunk 5, then chunk 2, then chunk 1, and seeing if chunk 1 matches the value that the verifier had already stored as the root of the file. note that the proof implicitly includes the index sometimes you need to add the partner chunk on the right before hashing and sometimes on the left, and if the index used to verify the proof is different then the proof will not match. thus, if i ask for a proof of piece 422, and you instead provide even a valid proof of piece 587, i will notice that something is wrong. also, there is no way to provide a proof without ownership of the entire relevant section of the merkle tree; if you try to pass off fake data, at some point the hashes will mismatch and the final root will be different. now, let's go over the protocol. i construct a merkle tree out of the file as described above, and upload this to some party. then, every 12 hours, i pick a random number in [0, 2^k-1] and submit that number as a challenge. if the storer replies back with a merkle tree proof, then i verify the proof and if it is correct send 0.001 btc (or eth, or storjcoin, or whatever other token is used). if i receive no proof or an invalid proof, then i do not send btc. if the storer stores the entire file, they will succeed 100% of the time, if they store 50% of the file they will succeed 50% of the time, etc. if we want to make it all-or-nothing, then we can simply require the storer to solve ten consecutive proofs in order to get a reward. the storer can still get away with storing 99%, but then we take advantage of the same redundant coding strategy that i mentioned above and will describe below to make 90% of the file sufficient in any case. one concern that you may have at this point is privacy if you use a cryptographic protocol to let any node get paid for storing your file, would that not mean that your files are spread around the internet so that anyone can potentially access them? fortunately the answer to this is simple: encrypt the file before sending it out. from this point on, we'll assume that all data is encrypted, and ignore privacy because the presence of encryption resolves that issue almost completely (the "almost" being that the size of the file, and the times at which you access the file, are still public). looking to decentralize so now we have a protocol for paying people to store your data; the algorithm can even be made trust-free by putting it into an ethereum contract, using block.prevhash as a source of random data to generate the challenges. now let's go to the next step: figuring out how to decentralize the storage and add redundancy. the simplest way to decentralize is simple replication: instead of one node storing one copy of the file, we can have five nodes storing one copy each. however, if we simply follow the naive protocol above, we have a problem: one node can pretend to be five nodes and collect a 5x return. a quick fix to this is to encrypt the file five times, using five different keys; this makes the five identical copies indistinguishable from five different files, so a storer will not be able to notice that the five files are the same and store them once but claim a 5x reward. but even here we have two problems. first, there is no way to verify that the five copies of the file are stored by five separate users. if you want to have your file backed up by a decentralized cloud, you are paying for the service of decentralization; it makes the protocol have much less utility if all five users are actually storing everything through google and amazon. this is actually a hard problem; although encrypting the file five times and pretending that you are storing five different files will prevent a single actor from collecting a 5x reward with 1x storage, it cannot prevent an actor from collecting a 5x reward with 5x storage, and economies of scale mean even that situation will be desirable from the point of view of some storers. second, there is the issue that you are taking a large overhead, and especially taking the false-redundancy issue into account you are really not getting that much redundancy from it for example, if a single node has a 50% chance of being offline (quite reasonable if we're talking about a network of files being stored in the spare space on people's hard drives), then you have a 3.125% chance at any point that the file will be inaccessible outright. there is one solution to the first problem, although it is imperfect and it's not clear if the benefits are worth it. the idea is to use a combination of proof of stake and a protocol called "proof of custody" proof of simultaneous possession of a file and a private key. if you want to store your file, the idea is to randomly select some number of stakeholders in some currency, weighting the probability of selection by the number of coins that they have. implementing this in an ethereum contract might involve having participants deposit ether in the contract (remember, deposits are trust-free here if the contract provides a way to withdraw) and then giving each account a probability proportional to its deposit. these stakeholders will then receive the opportunity to store the file. then, instead of the simple merkle tree check described in the previous section, the proof of custody protocol is used. the proof of custody protocol has the benefit that it is non-outsourceable there is no way to put the file onto a server without giving the server access to your private key at the same time. this means that, at least in theory, users will be much less inclined to store large quantities of files on centralized "cloud" computing systems. of course, the protocol accomplishes this at the cost of much higher verification overhead, so that leaves open the question: do we want the verification overhead of proof of custody, or the storage overhead of having extra redundant copies just in case? m of n regardless of whether proof of custody is a good idea, the next step is to see if we can do a little better with redundancy than the naive replication paradigm. first, let's analyze how good the naive replication paradigm is. suppose that each node is available 50% of the time, and you are willing to take 4x overhead. in those cases, the chance of failure is 0.5 ^ 4 = 0.0625 a rather high value compared to the "four nines" (ie. 99.99% uptime) offered by centralized services (some centralized services offer five or six nines, but purely because of talebian black swan considerations any promises over three nines can generally be considered bunk; because decentralized networks do not depend on the existence or actions of any specific company or hopefully any specific software package, however, decentralized systems arguably actually can promise something like four nines legitimately). if we assume that the majority of the network will be quasi-professional miners, then we can reduce the unavailability percentage to something like 10%, in which case we actually do get four nines, but it's better to assume the more pessimistic case. what we thus need is some kind of m-of-n protocol, much like multisig for bitcoin. so let's describe our dream protocol first, and worry about whether it's feasible later. suppose that we have a file of 1 gb, and we want to "multisig" it into a 20-of-60 setup. we split the file up into 60 chunks, each 50 mb each (ie. 3 gb total), such that any 20 of those chunks suffice to reconstruct the original. this is information-theoretically optimal; you can't reconstruct a gigabyte out of less than a gigabyte, but reconstructing a gigabyte out of a gigabyte is entirely possible. if we have this kind of protocol, we can use it to split each file up into 60 pieces, encrypt the 60 chunks separately to make them look like independent files, and use an incentivized file storage protocol on each one separately. now, here comes the fun part: such a protocol actually exists. in this next part of the article, we are going to describe a piece of math that is alternately called either "secret sharing" or "erasure coding" depending on its application; the algorithm used for both those names is basically the same with the exception of one implementation detail. to start off, we will recall a simple insight: two points make a line. particularly, note that there is exactly one line that passes through those two points, and yet there is an infinite number of lines that pass through one point (and an infinite number of lines that pass through zero points). out of this simple insight, we can make a restricted 2-of-n version of our encoding: treat the first half of the file as the y coordinate of a line at x = 1 and the second half as the y coordinate of the line at x = 2 , draw the line, and take points at x = 3 , x = 4 , etc. any two pieces can then be used to reconstruct the line, and from there derive the y coordinates at x = 1 and x = 2 to get the file back. mathematically, there are two ways of doing this. the first is a relatively simple approach involving a system of linear equations. suppose that we file we want to split up is the number "1321". the left half is 13, the right half is 21, so the line joins (1, 13) and (2, 21). if we want to determine the slope and y-intercept of the line, we can just solve the system of linear equations: subtract the first equation from the second, and you get: and then plug that into the first equation, and get: so we have our equation, y = 8 * x + 5. we can now generate new points: (3, 29), (4, 37), etc. and from any two of those points we can recover the original equation. now, let's go one step further, and generalize this into m-of-n. as it turns out, it's more complicated but not too difficult. we know that two points make a line. we also know that three points make a parabola: thus, for 3-of-n, we just split the file into three, take a parabola with those three pieces as the y coordinates at x = 1, 2, 3 , and take further points on the parabola as additional pieces. if we want 4-of-n, we use a cubic polynomial instead. let's go through that latter case; we still keep our original file, "1321", but we'll split it up using 4-of-7 instead. our four points are (1, 1) , (2, 3) , (3, 2) , (4, 1) . so we have: eek! well, let's, uh, start subtracting. we'll subtract equation 1 from equation 2, 2 from 3, and 3 from 4, to reduce four equations to three, and then repeat that process again and again. so a = 1/2. now, we unravel the onion, and get: so b = -9/2, and then: so c = 12, and then: so a = 0.5, b = -4.5, c = 12, d = -7. here's the lovely polynomial visualized: i created a python utility to help you do this (this utility also does other more advanced stuff, but we'll get into that later); you can download it here. if you wanted to solve the equations quickly, you would just type in: > import share > share.sys_solve([[1.0, 1.0, 1.0, 1.0, -1.0], [8.0, 4.0, 2.0, 1.0, -3.0], [27.0, 9.0, 3.0, 1.0, -2.0], [64.0, 16.0, 4.0, 1.0, -1.0]]) [0.5, -4.5, 12.0, -7.0] note that putting the values in as floating point is necessary; if you use integers python's integer division will screw things up. now, we'll cover the easier way to do it, lagrange interpolation. the idea here is very clever: we come up with a cubic polynomial whose value is 1 at x = 1 and 0 at x = 2, 3, 4, and do the same for every other x coordinate. then, we multiply and add the polynomials together; for example, to match (1, 3, 2, 1) we simply take 1x the polynomial that passes through (1, 0, 0, 0), 3x the polynomial through (0, 1, 0, 0), 2x the polynomial through (0, 0, 1, 0) and 1x the polynomial through (0, 0, 0, 1) and then add those polynomials together to get the polynomal through (1, 3, 2, 1) (note that i said the polynomial passing through (1, 3, 2, 1); the trick works because four points define a cubic polynomial uniquely). this might not seem easier, because the only way we have of fitting polynomials to points to far is the cumbersome procedure above, but fortunately, we actually have an explicit construction for it: at x = 1, notice that the top and bottom are identical, so the value is 1. at x = 2, 3, 4, however, one of the terms on the top is zero, so the value is zero. multiplying up the polynomials takes quadratic time (ie. ~16 steps for 4 equations), whereas our earlier procedure took cubic time (ie. ~64 steps for 4 equations), so it's a substantial improvement especially once we start talking about larger splits like 20-of-60. the python utility supports this algorithm too: > import share > share.lagrange_interp([1.0, 3.0, 2.0, 1.0], [1.0, 2.0, 3.0, 4.0]) [-7.0, 12.000000000000002, -4.5, 0.4999999999999999] the first argument is the y coordinates, the second is the x coordinates. note the opposite order here; the code in the python module puts the lower-order coefficients of the polynomial first. and finally, let's get our additional shares: > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 5) 3.0 > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 6) 11.0 > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 7) 28.0 so here immediately we can see two problems. first, it looks like computerized floating point numbers aren't infinitely precise after all; the 12 turned into 12.000000000000002. second, the chunks start getting large as we move further out; at x = 10, it goes up to 163. this is somewhat breaking the promise that the amount of data you need to recover the file is the same size as the original file; if we lose x = 1, 2, 3, 4 then you need 8 digits to get the original values back and not 4. these are both serious issues, and ones that we will resolve with some more mathematical cleverness later, but we'll leave them aside for now. even with those issues remaining, we have basically achieved victory, so let's calculate our spoils. if we use a 20-of-60 split, and each node is online 50% of the time, then we can use combinatorics specifically, the binomial distribution formula to compute the probability that our data is okay. first, to set things up: > def fac(n): return 1 if n==0 else n * fac(n-1) > def choose(n,k): return fac(n) / fac(k) / fac(n-k) > def prob(n,k,p): return choose(n,k) * p ** k * (1-p) ** (n-k) the last formula computes the probability that exactly k servers out of n will be online if each individual server has a probability p of being online. now, we'll do: > sum([prob(60, k, 0.5) for k in range(0, 20)]) 0.0031088013296633353 99.7% uptime with only 3x redundancy a good step up from the 87.5% uptime that 3x redundancy would have given us had simple replication been the only tool in our toolkit. if we crank the redundancy up to 4x, then we get six nines, and we can stop there because the probability either ethereum or the entire internet will crash outright is greater than 0.0001% anyway (in fact, you're more likely to die tomorrow). oh, and if we assume each machine has 90% uptime (ie. hobbyist "farmers"), then with a 1.5x-redundant 20-of-30 protocol we get an absolutely overkill twelve nines. reputation systems can be used to keep track of how often each node is online. dealing with errors we'll spend the rest of this article discussing three extensions to this scheme. the first is a concern that you may have skipped over reading the above description, but one which is nonetheless important: what happens if some node tries to actively cheat? the algorithm above can recover the original data of a 20-of-60 split from any 20 pieces, but what if one of the data providers is evil and tries to provide fake data to screw with the algorithm. the attack vector is a rather compelling one: > share.lagrange_interp([1.0, 3.0, 2.0, 5.0], [1.0, 2.0, 3.0, 4.0]) [-11.0, 19.333333333333336, -8.5, 1.1666666666666665] taking the four points of the above polynomial, but changing the last value to 5, gives a completely different result. there are two ways of dealing with this problem. one is the obvious way, and the other is the mathematically clever way. the obvious way is obvious: when splitting a file, keep the hash of each chunk, and compare the chunk against the hash when receiving it. chunks that do not match their hashes are to be discarded. the clever way is somewhat more clever; it involves some spooky not-quite-moon-math called the berlekamp-welch algorithm. the idea is that instead of fitting just one polynomial, p, we imagine into existence two polynomials, q and e, such that q(x) = p(x) * e(x), and try to solve for both q and e at the same time. then, we compute p = q / e. the idea is that if the equation holds true, then for all x either p(x) = q(x) / e(x) or e(x) = 0; hence, aside from computing the original polynomial we magically isolate what the errors are. i won't go into an example here; the wikipedia article has a perfectly decent one, and you can try it yourself with: > map(lambda x: share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], x), [1, 2, 3, 4, 5, 6]) [1.0, 3.0, 2.0, 1.0, 3.0, 11.0] > share.berlekamp_welch_attempt([1.0, 3.0, 18018.0, 1.0, 3.0, 11.0], [1, 2, 3, 4, 5, 6], 3) [-7.0, 12.0, -4.5, 0.5] > share.berlekamp_welch_attempt([1.0, 3.0, 2.0, 1.0, 3.0, 0.0], [1, 2, 3, 4, 5, 6], 3) [-7.0, 12.0, -4.5, 0.5] now, as i mentioned, this mathematical trickery is not really all that needed for file storage; the simpler approach of storing hashes and discarding any piece that does not match the recorded hash works just fine. but it is incidentally quite useful for another application: self-healing bitcoin addresses. bitcoin has a base58check encoding algorithm, which can be used to detect when a bitcoin address has been mistyped and returns an error so you do not accidentally send thousands of dollars into the abyss. however, using what we know, we can actually do better and make an algorithm which not only detects mistypes but also actually corrects the errors on the fly. we don't use any kind of clever address encoding for ethereum because we prefer to encourage use of name registry-based alternatives, but if an address encoding scheme was demanded something like this could be used. finite fields now, we get back to the second problem: once our x coordinates get a little higher, the y coordinates start shooting off very quickly toward infinity. to solve this, what we are going to do is nothing short of completely redefining the rules of arithmetic as we know them. specifically, let's redefine our arithmetic operations as: a + b := (a + b) % 11 a b := (a b) % 11 a * b := (a * b) % 11 a / b := (a * b ** 9) % 11 that "percent" sign there is "modulo", ie. "take the remainder of dividing that vaue by 11", so we have 7 + 5 = 1 , 6 * 6 = 3 (and its corollary 3 / 6 = 6 ), etc. we are now only allowed to deal with the numbers 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. the surprising thing is that, even as we do this, all of the rules about traditional arithmetic still hold with our new arithmetic; (a * b) * c = a * (b * c) , (a + b) * c = (a * c) + (b * c) , a / b * b = a if b != 0 , (a^2 b^2) = (a b)*(a + b) , etc. thus, we can simply take the algebra behind our polynomial encoding that we used above, and transplant it over into the new system. even though the intuition of a polynomial curve is completely borked we're now dealing with abstract mathematical objects and not anything resembling actual points on a plane because our new algebra is self-consistent, the formulas still work, and that's what counts. > e = share.mkmoduloclass(11) > p = share.lagrange_interp(map(e, [1, 3, 2, 1]), map(e, [1, 2, 3, 4])) > p [4, 1, 1, 6] > map(lambda x: share.eval_poly_at(map(e, p), e(x)), range(1, 9)) [1, 3, 2, 1, 3, 0, 6, 2] > share.berlekamp_welch_attempt(map(e, [1, 9, 9, 1, 3, 0, 6, 2]), map(e, [1, 2, 3, 4, 5, 6, 7, 8]), 3) [4, 1, 1, 6] the "map(e, [v1, v2, v3]) " is used to convert ordinary integers into elements in this new field; the software library includes an implementation of our crazy modulo 11 numbers that interfaces with arithmetic operators seamlessly so we can simply swap them in (eg. print e(6) * e(6) returns 3 ). you can see that everything still works except that now, because our new definitions of addition, subtraction, multiplication and division always return integers in [0 ... 10] we never need to worry about either floating point imprecision or the numbers expanding as the x coordinate gets too high. now, in reality these relatively simple modulo finite fields are not what are usually used in error-correcting codes; the generally preferred construction is something called a galois field (technically, any field with a finite number of elements is a galois field, but sometimes the term is used specifically to refer to polynomial-based fields as we will describe here). the idea is that the elements in the field are now polynomials, where the coefficients are themselves values in the field of integers modulo 2 (ie. a + b := (a + b) % 2, etc). adding and subtracting work as normally, but multiplying is itself modulo a polynomial, specifically x^8 + x^4 + x^3 + x + 1. this rather complicated multilayered construction lets us have a field with exactly 256 elements, so we can conveniently store every element in one byte and every byte as one element. if we want to work on chunks of many bytes at a time, we simply apply the scheme in parallel (ie. if each chunk is 1024 bytes, determine 10 polynomials, one for each byte, extend them separately, and combine the values at each x coordinate to get the chunk there). but it is not important to know the exact workings of this; the salient point is that we can redefine +, -, * and / in such a way that they are still fully self-consistent but always take and output bytes. going multidimensional: the self-healing cube now, we're using finite fields, and we can deal with errors, but one issue still remains: what happens when nodes do go down? at any point in time, you can count on 50% of the nodes storing your file staying online, but what you cannot count on is the same nodes staying online forever eventually, a few nodes are going to drop out, then a few more, then a few more, until eventually there are not enough of the original nodes left online. how do we fight this gradual attrition? one strategy is that you could simply watch the contracts that are rewarding each individual file storage instance, seeing when some stop paying out rewards, and then re-upload the file. however, there is a problem: in order to re-upload the file, you need to reconstruct the file in its entirety, a potentially difficult task for the multi-gigabyte movies that are now needed to satisfy people's seemingly insatiable desires for multi-thousand pixel resolution. additionally, ideally we would like the network to be able to heal itself without requiring active involvement from a centralized source, even the owner of the files. fortunately, such an algorithm exists, and all we need to accomplish it is a clever extension of the error correcting codes that we described above. the fundamental idea that we can rely on is the fact that polynomial error correcting codes are "linear", a mathematical term which basically means that it interoperates nicely with multiplication and addition. for example, consider: > share.lagrange_interp([1.0, 3.0, 2.0, 1.0], [1.0, 2.0, 3.0, 4.0]) [-7.0, 12.000000000000002, -4.5, 0.4999999999999999] > share.lagrange_interp([10.0, 5.0, 5.0, 10.0], [1.0, 2.0, 3.0, 4.0]) [20.0, -12.5, 2.5, 0.0] > share.lagrange_interp([11.0, 8.0, 7.0, 11.0], [1.0, 2.0, 3.0, 4.0]) [13.0, -0.5, -2.0, 0.5000000000000002] > share.lagrange_interp([22.0, 16.0, 14.0, 22.0], [1.0, 2.0, 3.0, 4.0]) [26.0, -1.0, -4.0, 1.0000000000000004] see how the input to the third interpolation is the sum of the inputs to the first two, and the output ends up being the sum of the first two outputs, and then when we double the input it also doubles the output. so what's the benefit of this? well, here's the clever trick. erasure cording is itself a linear formula; it relies only on multiplication and addition. hence, we are going to apply erasure coding to itself. so how are we going to do this? here is one possible strategy. first, we take our 4-digit "file" and put it into a 2x2 grid. 1 3 2 1 then, we use the same polynomial interpolation and extension process as above to extend the file along both the x and y axes: 1 3 5 7 2 1 0 10 3 10 4 8 and then we apply the process again to get the remaining 4 squares: 1 3 5 7 2 1 0 10 3 10 6 2 4 8 1 5 note that it doesn't matter if we get the last four squares by expanding horizontally and vertically; because secret sharing is linear it is commutative with itself, so you get the exact same answer either way. now, suppose we lose a number in the middle, say, 6. well, we can do a repair vertically: > share.repair([5, 0, none, 1], e) [5, 0, 6, 1] or horizontally: > share.repair([3, 10, none, 2], e) [3, 10, 6, 2] and tada, we get 6 in both cases. this is the surprising thing: the polynomials work equally well on both the x or the y axis. hence, if we take these 16 pieces from the grid, and split them up among 16 nodes, and one of the nodes disappears, then nodes along either axis can come together and reconstruct the data that was held by that particular node and start claiming the reward for storing that data. ideally, we can even extend this process beyond 2 dimensions, producing a 3-dimensional cube, a 4-dimensional hypercube or more the gain of using more dimensions is ease of reconstruction, and the cost is a lower degree of redundancy. thus, what we have is an information-theoretic equivalent of something that sounds like it came straight out of science-fiction: a highly redundant, interlinking, modular self-healing cube, that can quickly locally detect and fix its own errors even if large sections of the cube were to be damaged, co-opted or destroyed. "the cube can still function even if up to 78% of it were to be destroyed..." so, let's put it all together. you have a 10 gb file, and you want to split it up across the network. first, you encrypt the file, and then you split the file into, let's say, 125 chunks. you arrange these chunks into a 3-dimensional 5x5x5 cube, figure out the polynomial along each axis, and "extend" each one so that at the end you have a 7x7x7 cube. you then look for 343 nodes willing to store each piece of data, and tell each node only the identity of the other nodes that are along the same axis (we want to make an effort to avoid a single node gathering together an entire line, square or cube and storing it and calculating any redundant chunks as needed in real-time, getting the reward for storing all the chunks of the file without actually providing any redundancy. in order to actually retrieve the file, you would send out a request for all of the chunks, then see which of the pieces coming in have the highest bandwidth. you may use the pay-per-chunk protocol to pay for the sending of the data; extortion is not an issue because you have such high redundancy so no one has the monopoly power to deny you the file. as soon as the minimal number of pieces arrive, you would do the math to decrypt the pieces and reconstitute the file locally. perhaps, if the encoding is per-byte, you may even be able to apply this to a youtube-like streaming implementation, reconstituting one byte at a time. in some sense, there is an unavoidable tradeoff between self-healing and vulnerability to this kind of fake redundancy: if parts of the network can come together and recover a missing piece to provide redundancy, then a malicious large actor in the network can recover a missing piece on the fly to provide and charge for fake redundancy. perhaps some scheme involving adding another layer of encryption on each piece, hiding the encryption keys and the addresses of the storers of the individual pieces behind yet another erasure code, and incentivizing the revelation process only at some particular times might form an optimal balance. secret sharing at the beginning of the article, i mentioned another name for the concept of erasure coding, "secret sharing". from the name, it's easy to see how the two are related: if you have an algorithm for splitting data up among 9 nodes such that 5 of 9 nodes are needed to recover it but 4 of 9 can't, then another obvious use case is to use the same algorithm for storing private keys split up your bitcoin wallet backup into nine parts, give one to your mother, one to your boss, one to your lawyer, put three into a few safety deposit boxes, etc, and if you forget your password then you'll be able to ask each of them individually and chances are at least five will give you your pieces back, but the individuals themselves are sufficiently far apart from each other that they're unlikely to collude with each other. this is a very legitimate thing to do, but there is one implementation detail involved in doing it right. the issue is this: even though 4 of 9 can't recover the original key, 4 of 9 can still come together and have quite a lot of information about it specifically, four linear equations over five unknowns. this reduces the dimensionality of the choice space by a factor of 5, so instead of 2256 private keys to search through they now have only 251. if your key is 180 bits, that goes down to 236 trivial work for a reasonably powerful computer. the way we fix this is by erasure-coding not just the private key, but rather the private key plus 4x as many bytes of random gook. more precisely, let the private key be the zero-degree coefficient of the polynomial, pick four random values for the next four coefficients, and take values from that. this makes each piece five times longer, but with the benefit that even 4 of 9 now have the entire choice space of 2180 or 2256 to search through. conclusion so there we go, that's an introduction to the power of erasure coding arguably the single most underhyped set of algorithms (except perhaps scip) in computer science or cryptography. the ideas here essentially are to file storage what multisig is to smart contracts, allowing you to get the absolutely maximum possible amount of security and redundancy out of whatever ratio of storage overhead you are willing to accept. it's an approach to file storage availability that strictly supersedes the possibilities offered by simple splitting and replication (indeed, replication is actually exactly what you get if you try to apply the algorithm with a 1-of-n strategy), and can be used to encapsulate and separately handle the problem of redundancy in the same way that encryption encapsulates and separately handles the problem of privacy. decentralized file storage is still far from a solved problem; although much of the core technology, including erasure coding in tahoe-lafs, has already been implemented, there are certainly many minor and not-so-minor implementation details that still need to be solved for such a setup to actually work. an effective reputation system will be required for measuring quality-of-service (eg. a node up 99% of the time is worth at least 3x more than a node up 50% of the time). in some ways, incentivized file storage even depends on effective blockchain scalability; having to implicitly pay for the fees of 343 transactions going to verification contracts every hour is not going to work until transaction fees become far lower than they are today, and until then some more coarse-grained compromises are going to be required. but then again, pretty much every problem in the cryptocurrency space still has a very long way to go. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements what if ethereum lived on a treap? or, blockchains charging rent | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search what if ethereum lived on a treap? or, blockchains charging rent posted by vitalik buterin on may 27, 2014 research & development although truly solving blockchain scalability fundamentally, that is to say figuring out a solution to the problem that every node must process every transaction, is a very hard problem, and all suggested solutions rely on either highly advanced cryptography or intricate multi-blockchain architectures, partial solutions that provide a constant-factor improvement over the way bitcoin does things are actually quite easy to find. in ethereum, for example, we have the concept of a separate state tree and transaction history, allowing miners to easily store only existing account states and not historical transaction outputs that are no longer relevant and thereby drastically reducing the amount of storage that would be required; if bitcoin is any indication, savings should be around 90%. another improvement is the use of accounts instead of coins/utxo as the fundamental unit, allowing each user to take up less than a hundred bytes on the blockchain regardless of how many transactions go in and out of their account. of course, both of these are partially, or perhaps even fully, offset by the fact that ethereum has a much larger scope, intending to use the blockchain for much more than just monetary transactions, but even if that is true it makes scalability all the more necessary. what i am about to describe in this article is another anti-bloat strategy that could potentially be used to achieve very substantial gains, this time targeting the issue of “dust”. dust, in simple terms, refers to the accumulation of tiny outputs (or accounts) on the blockchain, perhaps with only a fraction of a cent worth of coin, that are either dumped onto the blockchain maliciously or are simply too low-value to be even worth the increased transaction fee to send. on ethereum, dust of the second kind can also consist of accounts that have zero balance left, perhaps because the user might want to switch to a different private key for security reasons. dust is a serious problem; it is estimated that the majority of the bitcoin blockchain is dust, and in the case of litecoin something like 90% of the outputs are the result of a single malicious blockchain spam attack that took place back to 2011. in ethereum, there is a storage fee onsstore in order to charge for adding something to the state, and the floating block limit system ensures that even a malicious miner has no significant advantage in this regard, but there is no concept of a fee charged over time; hence, there is no protection or incentive against a litecoin-style attack affecting the ethereum blockchain as well. but what if there was one? what if the blockchain could charge rent? the basic idea behind charging rent is simple. each account would keep track of how much space it takes up, including the [ nonce, balance, code, state_root ] header rlp and the storage tree, and then every block the balance would go down by rentfee multiplied by the amount of space taken up (which can be measured in bytes, for simplicity normalizing the total memory load of each storage slot to 64 bytes). if the balance of an account drops below zero, it would disappear from the blockchain. the hard part is implementation. actually implementing this scheme is in one way easier and in one way harder than expected. the easy part is that you do not need to actually update every account every block; all you do is keep track of the last block during which the account was manipulated and the amount of space taken up by the account in the header rlp and then read just the account every time computation accesses it. the hard part, however, is deleting accounts with negative balance. you might think that you can just scan through all accounts from time to time and then remove the ones with negative balances from the database; the problem is, however, that such a mechanism doesn’t play nicely with patricia trees. what if a new user joins the network at block 100000, wants to download the state tree, and there are some deleted accounts? some nodes will have to store the deleted accounts to justify the empty spots, the hashes corresponding to nothing, in the trie. what if a light client wants a proof of execution for some particular transaction? then the node supplying the proof will have to include the deleted accounts. one approach is to have a “cleansing block” every 100000 blocks that scans through the entire state and clears out the cruft. however, what if there was a more elegant solution? treaps one elegant data structure in computer science is something called a treap. a treap, as one might or probably might not understand from the name, is a structure which is simultaneously a tree and a heap. to review the relevant data structure theory, a heap) is a binary tree, where each node except for leaves has one or two children, where each node has a lower value than its children and the lowest-value node is at the top, and what data structure theorists normally call a tree is a binary tree where values are arranged in sorted order left to right (ie. a node is always greater than its left child and less than its right child, if present). a treap combines the two by having nodes with both a key and a priority; the keys are arranged horizontally and the priorities vertically. although there can be many heaps for each set of priorities, and many binary trees for each set of values, as it turns out it can be proven that there is always exactly one treap that matches every set of (priority, value)pairs. also, as it turns out, there is an easy (ie. log-time) algorithm for adding and removing a value from the treap, and the mathematical property that there is only one treap for every set of (priority, value) pairs means that treaps are deterministic, and both of these things together make treaps a potential strong candidate for replacing patricia trees as the state tree data structure. but then, the question is, what would we use for priorities? the answer is simple: the priority of a node is the expected block number at which the node would disappear. the cleaning process would then simply consist of repeatedly kicking off nodes at the top of the treap, a log-time process that can be done at the end of every block. however, there is one implementation difficulty that makes treaps somewhat challenging for this purpose: treaps are not guaranteed to be shallow. for example, consider the values [[5, 100], [6, 120], [7, 140], [8, 160], [9, 180]]. the treap for those would unfortunately look like this: now, imagine that an attacker generates ten thousand addresses, and puts them into sorted order. the attacker then creates an account with the first private key, and gives it enough ether to survive until block 450000. the attacker then gives the second private key enough ether to survive until block 450001. the third private key lasts until 450002, and so forth until the last account susrvives until block 459999. all of these go into the blockchain. now, the blockchain will have a chain of ten thousand values each of which is below and to the right of all of the previous. now, the attacker starts sending transactions to the addresses in the second half of the list. each of those transactions will require ten thousand database accesses to go through the treap to process. basically, a denial of service attack through trie manipulation. can we mitigate this by having the priorities decided according to a more clever semi-randomized algorithm? not really; even if priorities were completely random, there is an algorithm using which the attacker would be able to generate a 10000-length subsequence of accounts that have both address and priority in increasing order in a hundred million steps. can we mitigate this by updating the treap bottom-up instead of top-down? also no; the fact that these are merkle trees means that we basically have to use functional algorithms to get anywhere. so what can we do? one approach is to figure out a way to patch this attack. the simplest option would likely involve having a higher cost to purchasing priority the more levels you go down the tree. if the treap is currently 30 levels deep but your addition would increase it to 31 levels, the extra level would be a cost that must be paid for. however, this requires the trie nodes to include a built-in height variable, making the data structure somewhat more complicated and less minimalistic and pure. another approach is to take the idea behind treaps, and create a data structure that has the same effect using plain old boring patricia trees. this is the solution that is used in databases such as mysql, and is called “indices“. basically, instead of one trie we have two tries. one trie is a mapping of address to account header, and the other trie is a mapping of time-to-live to address. at the end of every block, the left side of the ttl trie is scanned, and as long as there are nodes that need to be deleted they are repeatedly removed from both tries. when a new node is added it is added to both tries, and when a node is updated a naive implementation would update it in both tries if the ttl is changed as a result of the transaction, but a more sophisticated setup might be made where the second update is only done in a more limited subset of cases; for example, one might create a system where a node needs to “purchase ttl” in blocks of 90 days, and this purchase happens automatically every time a node gets onto the chopping block – and if the node is too poor then of course it drops off the edge. consequences so now we have three strategies: treaps with heights, tries with time-to-live indices and the “cleansing block”. which one works best is an empirical question; the ttl approach would arguably be the easiest to graft onto existing code, but any one of the three could prove most effective assuming the inefficiencies of adding such a system, as well as the usability concerns of having disappearing contracts, are less severe than the gains. what would the effects of any of these strategies be? first of all, some contracts would need to start charging a micro-fee; even passive pieces of code like an elliptic curve signature verifier would need to continually spend funds to justify their existence, and those funds would have to come from somewhere. if a contract cannot afford to do this, then the contract could just store a hash and the onus would be on the transaction sender to send the contract the code that it is supposed to execute; the contract would then check the hash of the code and if the hash matches the code would be run. name-registry applications might decide to work somewhat differently, storing most of their registrations using some merkle tree-based offchain mechanism in order to reduce their rent. however, there is also another more subtle consequence: account nonce resets. for example, suppose that i have an account, and i received and sent some transactions from that account. in order to prevent replay attacks (ie. if i send 10 eth to bob, bob should not be able to republish the same transaction in order to get another 10 eth), each transaction includes a “nonce” counter that increments after every transaction. thus, the account header stores the current transaction nonce, and if the current nonce is 2 then the only transaction that will be accepted is one with a nonce of 2, at which point the nonce will go up to 3. if accounts disappear, then nonces could reset to 0, leading to potentially dangerous situations if a user accumulates some funds in an account, then lets the balance drop to zero and the account disappear, and then refills it. one solution would be for transactions to have a maximum block number, which can be set to 10 days in the future by defauly, and then require all withdrawals to leave enough balance for the account to last another 10 days; this way, old transactions with nonce 0 would be too old to replay. however, this adds another inefficiency, and must be balanced with the benefit of blockchains charging rent. as another interesting point, the history of the blockchain would become relevant again; some dapps, wishing to store some data forever, would store it in a transaction instead of the state, and then use past block headers as an immutable rent-free datastore. the existence of applications which do this would mean that ethereum clients would have to store at least a headers-only version of the history, compromising ethereum’s “the present state is all that matters” ideology. however, an alternative solution might be to have a contract maintaining a merkle mountain range, putting the responsibility onto those users that benefit from particular pieces of information being stored to maintain log-sized merkle tree proofs with the contract remaining under a kilobyte in size. as a final objection, what if storage space is not the most problematic point of pressure with regard to scalability? what if the main issue is with bandwidth or computation? if the problem is computation, then there are some convenient hacks that can be made; for example, the protocol might be expanded to include both transactions and state transition deltas into the block, and nodes would be free to only check a portion of the deltas (say, 10%) and then quickly gossip about inconsistencies to each other. if it’s bandwidth, then the problem is harder; it means that we simply cannot have every node downloading every transaction, so some kind of tree-chains solution is the only way to move forward. on the other hand, if space is the problem, then rent-charging blockchains are very likely the way to go. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements what is ethereum? project, platform, fuel, stack. | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search what is ethereum? project, platform, fuel, stack. posted by joseph lubin on may 14, 2014 research & development what is ethereum, the project? the ethereum project is an open source, community-driven effort designed to create a next-generation distributed application platform intended to be maximally flexible and powerful in the possibilities that it enables. what is ethereum, the platform? the ethereum platform combines a generalized peer-to-peer networking platform with next-generation blockchain architecture to deliver a decentralized consensus-based (decentcon), full-stack platform for developing, offering and using distributed application services.  a consumer-facing application, called the etherbrowser, integrates the front and back ends to create an environment in which anyone can easily and rapidly build highly secure, scalable and interoperable decentralized applications. like the bittorrent content sharing system, ethereum network nodes will run on thousands of computers around the world and, short of shutting down the internet, its operations cannot be halted.  this is because peer-to-peer systems generally involve a very large number of independent actors (people or organizations) each running the peer node software on one or more computers.  in the bitcoin system, these nodes are called “miners.” like bitcoin, in ethereum, the nodes on the network serve as miners whose purpose is to process and validate the transactions and computations on the system and quickly achieve a consensus regarding what happened on the system and when.  this consensus is what provides the network with its security.  the larger the number of nodes there are and the more work these nodes have to do to exercise a vote regarding what transpired on the network, the greater is the sense that this shared consensus view of the history of the system is a canonical and irrepudiable representation.  in ethereum, miners receive a reward for doing the work required to inform and enable their vote and they are also paid for providing resources to the network in the form of bandwidth, storage and computational processing. bitcoin is a system for securely transmitting and storing value.  as such, it can serve as the financial bedrock of the emerging global decentcon economy.  a conservative, prudent, development roadmap for bitcoin will make it easier to use and further secure the protocol against quirks and edge cases that might be exploited in the future (though it has thus far proven to be remarkably solid at the protocol level).  in contrast, as a platform for hosting distributed or decentralized applications (ðapps — spelled with the capital letter “eth” and pronounced “dapps” or “eth-apps” by the cognoscenti :-) ) and services, ethereum must be agile and forward moving.  in q4 of 2014, the ethereum team will deliver a feature-rich system that will be fully functional and will possess a rich user interface and deliver a compelling user experience for both end users and businesses building ðapps and offering services on the platform.  but technology moves fast, so ethereum will require an upgrade roadmap and continual development. what is ether, the cryptofuel? just as the bitcoin system has a token, called a bitcoin (lower case) that serves as the medium of exchange, ethereum has ether (eth) which serves as a unit of exchange to some degree, but more importantly, it serves as a fuel that powers applications on the ethereum system. the engineers of the ethereum project are building a computational device or appliance, in the form of a software program, that anyone can download and run on their computer, smart phone, or on dedicated, fast hardware.  in order to operate this software appliance a certain type of token is required as fuel in appropriate quantities. distributed applications on ethereum require payments of this token to fuel every computational and storage operation on the system. without requiring payments for operations, the system would be vulnerable to many sorts of attacks and would not be viable or secure. the payments are made to owners of computational resources in exchange for securing the ethereum network, for transmitting transactions, for storing data and for processing computations required by distributed software applications. people and businesses are interested in purchasing eth to power their own business applications, to make use of business applications offered by other service providers, to trade on forthcoming exchanges, or to speculatively hold for future sale to people and businesses.  eth may be purchased in the genesis sale (details forthcoming, please watch this space), on forthcoming 3rd-party exchanges and atms, and on exchanges that are implemented as dapps on ethereum. when purchasing eth in the genesis sale, the buyer is supporting the development of the product, just as with a kickstarter campaign. when the product is completed and ready for delivery, buyers will be able to claim their purchased eth from the genesis block — the root block of the ethereum blockchain. what is ethereum, the software stack? a software stack is a set of technologies, realized at different layers and levels of abstraction and possessing different complementary capabilities that work well together to enable a software development team to build a full, back-end to front-end software service for an end user.  ethereum provides a full-stack solution for developing and delivering ðapps, the front end of which may be accessed by an end user from a web page, dedicated front-end applications, or more commonly from the ethereum ðapp browser.  the ethereum stack is the first of its kind to enable developers to deliver decentcon software applications. when delivering an ðapp the developer or deployer of that ðapp does not arrange hosting of back-end server processes as with traditional software services.  rather, the code is embedded as payload in a transaction on the ethereum network and sent to one or more mining nodes on the network.  miners that receive the transaction will broadcast it to all of the peers that they are aware of, provided the transaction sender has included enough eth, the cryptofuel that powers operations on the system, to pay for the transaction.  this transaction diffuses through the network, peer to peer, and eventually is packaged into a block and locked into the blockchain.  blocks are created by miners approximately once per minute.  once the transaction with the code payload is embedded into a block, subsequent transactions can be sent to an address that is designated as the controller interface for that ðapp to invoke processing of the ðapp. when an end user wishes to activate one or more services offered by this ðapp, she will typically interact with a front-end user interface, loaded into the ðapp browser (probably qt-based or javascript/html5/css3) to trigger desired actions.  user interface components will be cached on some kind of decentralized bittorrent-like cloud and pulled in by the ðapp browser as needed. the user interface will formulate ethereum transactions and send these, with a suitable amount of the cryptofuel and any input data required, to the address of the controller interface of the ðapp.  once collected into a block, the transaction will trigger execution of the ðapp and the states of various components (called contracts) of the ðapp will transition to a result state.  a peer-to-peer fast messaging protocol, will enable the user interface to reflect such changes, and will facilitate communication among different dapps and among users. one thing to notice is that, from the perspective of application hosting, there is virtually nothing to be done.  the back end is launched into the blockchain “cloud” and the front end will usually be represented as a installable tile in the ethereum ðapp browser.  the end user will download the browser once and the browser will receive continual updates from bittorrent or a bittorrent-like distribution system.  while browsing the distributed ðapp catalog in the browser, the end user can install any ðapp of interest in her browser with a no cost, one-click installation.  these tiles will be organized categorically by users and each will be invokable into in a full blown, user interface “responsively” sized and configured to the dimensions and capabilities of the browser (some weaker devices may have limitations). the programming paradigm just described will be very unusual in comparison to typical development technologies and will require innovative (and perhaps sometimes kludgey) approaches.  if a programmer can only expect the state of the business logic to update once per minute, stochastically, techniques will have to be developed for certain kinds of applications to perhaps cache certain expected state changes and wait on back end processing before updating the front end.  even more complicated is the fact that a block housing transactions and associated ðapp state changes can be constructed and included in the blockchain but then find itself not part of the main consensus blockchain soon after and possibly have the associated transactions remain unconfirmed and unprocessed for a period of time.  worse, an intervening transaction might be processed first, thus rendering the first transaction invalid.  an entire new field of blockchain-based software development techniques is required.  many developers will forge novel solutions.  and some fundamentally new approaches may be required.  and for this, we are developing the crypto currency research group (ccrg) to conduct general research of benefit to the entire niche.  please watch this space for more on the ccrg. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum and oracles | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum and oracles posted by vitalik buterin on july 22, 2014 research & development one of the more popular proposals for implementing smart contracts differently from the way they are typically presented in ethereum is through the concept of oracles. essentially, instead of a long-running contract being run directly on the blockchain, all funds that are intended to go into the contract would instead go into an m-of-n multisig address controlled by a set of specialized entities called "oracles", and the contract code would be simultaneously sent to all of these entities. every time someone wants to send a message to the contract, they would send the message to the oracles. the oracles would run the code, and if the code execution leads to a withdrawal from the contract to some particular address then the oracles circulate a transaction sending the funds and sign it. the approach is still low-trust, as no single oracle has the ability to unilaterally withdraw the funds, but it has a number of particular advantages: not every node in the blockchain needs to perform the computation only a small number of oracles do it theoretically does not require as a platform anything more complicated than bitcoin or ripple as they currently stand contracts have a somewhat higher degree of privacy although exit transactions are still all visible, internal computations may not be. the scheme can also be augmented with secure multiparty computation protocols so the contract can even contain private information (something that would take efficient and secure obfuscation to work directly on ethereum) contracts can rely on external information (eg. currency prices, weather) since it is much easier for n nodes to come to consensus on the result of an http request than an entire blockchain. in fact, they can even rely on data from proprietary apis, if the oracles subscribe to the apis and pass along the costs to the contract users. given all of these advantages, it is undeniably clear that oracles have the potential to be a very useful paradigm for smart contracts going forward. however, the key question is, how will oracle-based computation and blockchain-based computation, as in ethereum, interact with each other? oracles are not always better first of all, one important point to make is that it will not always be the case that the oracle-based method of contract execution will be more efficient than the blockchain-based approach (not to mention non-currency/non-contract uses of the blockchain such as name registries and the people's republic of doug where oracle systems do not even begin to apply). a common misconception is that the primary feature of ethereum is that it is turing-complete, and so while bitcoin only allows quick scripts for verification ethereum contracts are means to do much harder and computationally intensive tasks. this is arguably a misconception. the primary feature of ethereum is not turing-completeness; in fact, we have a section in our whitepaper which makes the argument that even if we explicitly removed the ability of ethereum contracts to be turing-complete it would actually change very little and there would still be a need for "gas". in order to make contracts truly statically analyzable, we would need to go so far as to remove the first-class-citizen property (namely, the fact that contracts can create and call other contracts), at which point ethereum would have very limited utility. rather, the primary feature of ethereum is state ethereum accounts can contain not just a balance and code, but also arbitrary data, allowing for multi-step contracts, long-running contracts such as dos/dacs/daos and particularly non-financial blockchain-based applications to emerge. for example, consider the following contract: init: contract.storage[0] = msg.data[0] # limited account contract.storage[1] = msg.data[1] # unlimited account contract.storage[2] = block.timestamp # time last accessed code: if msg.sender == contract.storage[0]: last_accessed = contract.storage[2] balance_avail = contract.storage[3] # withdrawal limit is 1 finney per second, maximum 10000 ether balance_avail += 10^15 * (block.timestamp last_accessed) if balance_avail > 10^22: balance_avail = 10^22 if msg.data[1] <= balance_avail: send(msg.data[0], msg.data[1]) contract.storage[3] = balance_avail msg.data[1] contract.storage[2] = block.timestamp # unlimited account has no restrictions elif msg.sender == contact.storage[1]: send(msg.data[0], msg.data[1]) this contract is pretty straightforward. it is an account with two access keys, where the first key has a withdrawal limit and the second key does not. you can think of it as a cold/hot wallet setup, except that you do not need to periodically go to the cold wallet to refill unless you want to withdraw a large amount of ether all at once. if a message is sent with data [dest, value], then if the sender is the first account it can send up to a certain limit of ether, and the limit refills at the rate of 1 finney per second (ie. 86.4 ether per day). if the sender is the second account, then the account contract sends the desired amount of ether to the desired destination with no restrictions. now, let's see what expensive operations are required to execute here, specifically for a withdrawal with the limited key: an elliptic curve verification to verify the transaction 2 storage database reads to get the last access time and last withdrawable balance 1 storage database write to record the balance changes that result from the sending transaction 2 storage database writes to write the new last access time and withdrawable balance there are also a couple dozen stack operations and memory reads/writes, but these are much faster than database and cryptography ops so we will not count them. the storage database reads can be made efficient with caching, although the writes will require a few hashes each to rewrite the patricia tree so they are not as easy; that's why sload has a gas cost of 20 but sstore has a cost of up to 200. additionally, the entire transaction should take about 160 bytes, the serpent code takes up 180 bytes, and the four storage slots take up 100-150 bytes hence, 350 bytes one-time cost and 160 bytes bandwitdh per transaction. now, consider this contract with a multisig oracle. the same operations will need to be done, but only on a few servers so the cost is negligible. however, when the multisig transaction is sent to bitcoin, if the multisig is a 3-of-5 then three elliptic curve verifications will be required, and the transaction will require 65 bytes per signature plus 20 bytes per public key so it will take about 350-400 bytes altogether (including also metadata and inputs). the blockchain storage cost will be around 50 bytes per utxo (as opposed to a static 350 in ethereum). hence, assuming that an elliptic curve verification takes longer than a few hashes (it does), the blockchain-based approach is actually easier. the reason why this example is so favorable is because it is a perfect example of how ethereum is about state and not turing-completeness: no loops were used, but the magic of the contract came from the fact that a running record of the withdrawal limit could be maintained inside the contract. (note: advanced cryptographers may note that there is a specialized type of threshold signature that actually requires only one verification operation even if a large number of oracles are used to produce it. however, if we use a currency with such a feature built-in, then we are already abandoning bitcoin's existing infrastructure and network effect; in that case, why not just use the ethereum contract?) but sometimes they are at other times, however, oracles do make sense. the most common case that will appear in reality is the case of external data; sometimes, you want a financial contract that uses the price of the us dollar, and you can't cryptographically determine that just by doing a few hashes and measuring ratios. in this case, oracles are absolutely necessary. another important case is smart contracts that actually are very hard to evaluate. for example, if you are purchasing computational resources from a decentralized cloud computing application, verifying that computations were done legitimately is not a task that the ethereum blockchain can cheaply handle. for most classes of computation, verifying that they were done correctly takes exactly as long as doing them in the first place, so the only way to practically do such a thing is through occasional spot-checking using, well, oracles. another cloud-computing use case for oracles, although in this context we do not think of them as such, is file storage you absolutely do not want to back up your 1gb hard drive onto the blockchain. an additional use-case, already mentioned above, is privacy. sometimes, you may not want the details of your financial contracts public, so doing everything on-chain may not be the best idea. sure, you can use standard-form contracts, and people won't know that it's you who is making a contract for difference between eth and usd at 5:1 leverage, but the information leakage is still high. in those cases, you may want to limit what is done on-chain and do most things off-chain. so how can they work together so we have these two paradigms of total on-chain and partial on-chain, and they both have their relative strengths and weaknesses. however, the question is, are the two really purely competitive? the answer is, as it turns out, no. to further this point, here are a few particular examples: schellingcoin incentivized decentralized oracles. the schellingcoin protocol is a proof-of-concept that shows how we can create a decentralized oracle protocol that is incentive-compatible: have a two-step commitment protocol so that oracles do not initially know what each other's answers are, and then at the end have an ethereum contract reward those oracles that are closest to the median. this incentivizes everyone to respond with the truth, since it is very difficult to coordinate on a lie. an independently conceived alternative, truthcoin, does a similar thing for prediction markets with binary outcomes (eg. did the toronto maple leafs win the world cup?). verifiable computation oracles when the oracles in question are executing moderately computationally intensive code, then we can actually go beyond the admittedly flaky and untested economics of the schellingcoin/truthcoin protocols. the idea is as follows. by default, we have m of n oracles running the code and providing their votes on the answers. however, when an oracle is perceived to vote incorrectly, that oracles can be "challenged". at that point, the oracle must provide the code to the blockchain, the blockchain checks the code against a pre-provided hash and runs the code itself, and sees if the result matches. if the result does not match, or if the oracle never replies to the challenge, then it loses its security deposit. the game-theoretic equilibrium here is for there to be no cheating at all, since any attempt at cheating necessarily harms some other party and so that party has the incentive to perform a check. signature batching one of the problems that i pointed out with the multisig oracle approach above is signature bloat: if you have three oracles signing everything, then that's 195 extra bytes in the blockchain and three expensive verification operations per transaction. however, with ethereum we can be somewhat more clever we can come up with a specialized "oracle contract", to which oracles can submit a single transaction with a single signature with a large number of votes batched together: [addr1, vote1, addr2, vote2 ... ]. the oracle contract then processes the entire list of votes and updates all of the multisig voting pools contained inside it simultaneously. thus, one signature could be used to back an arbitrarily large number of votes, reducing the scalability concerns substantially. blockchain-based auditing the concept of oracle-based computation can actually go much further than the "bitcoin multisig oracle" (or, for that matter, ethereum multisig oracle) idea. the extreme is an approach where oracles also decide the one thing that the bitcoin-based schemes still leave the blockchain to decide: the order of transactions. if we abandon this requirement, then it is possible to achieve much higher degrees of efficiency by having an oracle maintain a centralized database of transactions and state as they come, providing a signed record of each new balance sheet as a transaction is applied, allowing for applications like microtransactions and high-frequency trading. however, this has obvious trust-problems; particularly, what if the oracle double-spends? fortunately, we can set up an ethereum contract to solve the problem. much like the verifiable computation example above, the idea is that by default everything would run entirely on the oracle, but if the oracle chooses to sign two different balance sheets that are the result of incompatible transactions then those two signatures can be imported into ethereum, and the contract will verify that those two signatures are valid, and if they are the contract will take away the oracle's security deposit. more complicated schemes to deal with other attack vectors are also possible. verifiable secure multiparty computation in the case where you are using oracles specifically for the purpose of maintaining private data, you can set up a protocol where the oracles securely choose a new secret key using multiparty random number generation every 24 hours, sign a message with the old key to prove to the world that the new key has authority, and then have to submit all of the computations that they made using the old key to the ethereum blockchain for verification. the old key would be revealed, but it would be useless since a message transferring ownership rights to the new key is already in the blockchain several blocks before. any malfeasance or nonfeasance revealed in the audit would lead to the loss of a security deposit. the larger overarching point of all this is that the primary raison d'être of ethereum is not just to serve as a smart contract engine; it is more generally to serve as a world-wide trust-free decentralized computer, albeit with the disadvantages that it can hold no secrets and it is about ten thousand times slower than a traditional machine. the work in developing cryptoeconomic protocols to ensure that ordinary people have access to reliable, trustworthy and efficient markets and institutions is not nearly done, and the most exciting end-user-centric innovation is likely what will be built on top. it is entirely possible to have systems which use ethereum for one thing, an m-of-n oracle setup for another thing, and some alternative network like maidsafe for something else; base-level protocols are your servant, not your master. special thanks to vlad zamfir for some of the ideas behind combining oracles and ethereum previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements bootstrapping a decentralized autonomous corporation, part 3: identity corp | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search bootstrapping a decentralized autonomous corporation, part 3: identity corp posted by vitalik buterin on december 31, 2013 research & development in the first two parts of this series, we talked about what the basic workings of a decentralized autonomous corporation might look like, and what kinds of challenges it might need to deal with to be effective. however, there is still one question that we have not answered: what might such corporations be useful for? bitcoin developer jeff garzik once suggested that one application migh be a sort of decentralized dropbox, where users can upload their files to a resilient peer-to-peer network that would be incentivized to keep those files reliably backed up. but aside from this particular example, what other applications might there be? what are the industries where decentralized corporations will not simply be a gimiick, but will rather be able to survive on their own merits and provide genuine value to society? arguably, there are three major categories where this is the case. first, there are the natural monopolies. for certain kinds of services, it simply makes no sense to have many hundreds of competing offerings all working at the same time; software protocols, languages and to some extent social networks and currencies all fit into this model. however, if the providers of these services are not held in check by a competitive market, the question is, who does hold them in check? who ensures that they charge a fair market price for their services, and do not set monopoly prices thousands of times above what the product actually costs to produce? a decentralized corporation can theoretically be designed so that no one involved in the price-setting mechanism has any such incentive. more generally, decentralized corporations can be made invulnerable to corruption in ways unimaginable in human-controlled system, although great care would certainly need to be taken not to introduce other vulnerabilities instead; bitcoin itself is a perfect example of this. second, there are services that violate government laws and regulations; the use of decentralized file-sharing networks for copyright infringement, and to a much lesser extent the use of bitcoin on sites like silk road, are both examples. as satoshi nakamoto put it, “governments are good at cutting off the heads of a centrally controlled networks like napster, but pure p2p networks like gnutella and tor seem to be holding their own.” finally, there are those cases where a decentralized network can simply maintain itself more efficiently and provides better services than any centralized alternative; the peer-to-peer network used by blizzard to distribute updates to its massively multiplayer online game world of warcraft is perhaps one of the purest examples. the rest of this article will outline one particular idea for a decentralized corporation that can potentially open up a number of new possibilities in cryptocurrency, creating designs that have vastly different properties from the cryptocurrencies we see today while still staying close to the cryptocurrency ideal. the basic concept is this: identity corp, a corporation whose sole purpose is to create cryptographically secure identity documents for individuals that they could sign messages with, and are linked to individuals’ physical identities. what’s the point? at first, the idea of creating yet another way to track people’s identity seems silly. here we are, having escaped the shackles of state-backed fiat currency and its onerous anti-money-laundering identity verification requirements and gotten into the semi-anonymous world of bitcoin, and i’m suggesting that we bring identity verification right back to the table? but of course, the choice between “nymity” and anonymity is not nearly quite so simple. even individuals facing potential lifetime imprisonment, such as silk road founder dread pirate roberts, still tend to maintain some kind of identity – in the aforementioned case, the identity is “dread pirate roberts” itself. why does he (or perhaps she, we may never know) do that? the answer is simple: he is also running a multimillion dollar business – namely, the online anonymous marketplace silk road, and he needs to provide customers some reassurance that he can be trusted. legal and even semi-legal businesses often show themselves in public, deliberately making themselves vulnerable to both government prosecution and harassment of varying degrees from disaffected customers. why do that? to show to the world that they now have an extra incentive to act honestly. the “crypto” in cryptography does come from the greek word for hiding, but in reality cryptography is often about verifying your identity as it is about concealing it. however, the sort of “identity” used by dread pirate roberts is different from the identity we are talking about here. the function of standard public key cryptographic identity is a limited one: to provide proof that two messages were created (or at least signed) by the same entity. this definition may seem strange at first; usually, we think of identities as determining “who someone is”. in reality, however, just like in the principle of relativity in physics, in the context of identity and reputation theory there is no “preferred frame” for determining which set of observations of a person constitute that core person, or if a person has multiple names which name is his or her “real name”. if i write articles as “vitalik buterin”, but make internet posts as “djargon135″, it is equally legitimate to say “djargon135 is actually vitalik buterin” as it is to say “vitalik buterin is actually djargon135″; in either case, what matters is that one set of messages claimed to be written by djargon135, and another set of messages claimed to be written by vitalik buterin, in fact have a common author. under this framework, a “real name” is distinguished from a “pseudonym” in one way and one way only: each entity can only have one real name. that is to say, while pseudonyms can be used to prove that two messages were created by the same entity, real names can also be used to prove that two messages were created by two different entities. but this still does not answer the question: why have real names at all? in fact, nearly all applications of a real name can be reduced to one fundamental concept: the giveaway. we all understand what a giveaway is: perhaps a corporation wishes to hand out a free sample of a product to attract potential customers, perhaps a homeless shelter with limited resources wants to feed everyone enough to survive, and thus not let anyone take triple portions for themselves, or perhaps a government agency administering a welfare program wants to prevent people from claiming welfare twice. the idea is simple: x units of some product, service or commodity per person, and if you want more you will have to get your second portion through other channels. one of the use cases of a “real name” used earlier, that of a company owner publishing his details to reassure customers that he is vulnerable to prosecution by law enforcement, does not look like an example of a giveaway, but in fact that company owner is a recipient of a particularly special kind of giveaway in society: that of reputation. in a public key reputation environment, an identity can be created at no cost, so everyone starts out with zero reputation, making business difficult at first. in a real-name system, however, everyone immediately starts out with one pre-made identity, and no way to acquire more, making that identity “expensive” and thus giving them a fixed quantity of reputation to start out with. instead of one free sample per person, it’s one free reputation per person, but the principle is the same. how to implement it actually implemening a system, of course, is a challenge. it is very difficult to do with any purely over-the-internet mechanism because anyone can trivially create multiple identites and make them all act like different people. it is certainly possible to weed out some fraud by applying statistical analysis on the messages that everyone signs (eg. if two different identities both consistently spell “actualy” instead of “actually”, that is some strong evidence that they might be linked); however, this can easily be circumvented by combining a spellchecker with a program that deliberately inserts spelling errors and rearranges some grammatical constructions. these tactics can perhaps be themselves corrected for, but ultimately relying solely or even largely on such mechanisms is a recipe for statistical warfare, not any kind of stable identity system. so what’s left? offline mechanisms. dna-based identity is the most obvious, although face, iris and fingerprint scans can also add themselves to the list. currently, government-based identity systems do not use this information too much because government identity documents follow a centralized parent-child model: you want a social insurance number, you need to provide your passport, you lost your passport, you provide a birth certificate and possibly change-of-name certificates if applicable. ultimately, everything usually depends on a combination of the birth certificate and face recognition on the part of he government agents administering the system. a decentralized system to accomplish this can use both mechanisms, although many will argue that having the ability in theory to register without providing any government documents is a strong positive – it should be possible to get an identity through the system without necessarily tying in one’s government-backed “real name” (in the usual sense of the term, not my own distinction given above). if this is not possible, then some kind of mixnet-like setup could be used to anonymize identities once they have been created while still maintaining the one-per-person limit. however, attempts at fraud would likely be much more frequent; governments are not, at least at first, going to use any legal mechanisms to enforce anti-fraud rules with these identities as they do with their own documents. from the above information, it becomes easy to imagine how one might create a centralized organization that accomplishes this objective. the organization would have an office, people would go in, have their biometrics (face, fingerprint, iris, maybe dna) checked, and would then receive their fresh new cryptographic passport. why not stop there? in this case, the answer is that the natural monopoly argument applies. even if the system may have multiple identity providers, they would all need to cross-check information with each other to prevent multiple signups, and the resulting system would necessarily be the only one of its kind. if this system is managed by a corporation, that corporation would have the incentive to start charging high fees once its product becomes ubiquitous and necessary. if it is managed by a government, then the government would have the incentive to tie these identities to its own real names, and remove any privacy features (or at least install a backdoor for itself). furthermore, it might want the ability to revoke identities as a punishment, and if large parts of the internet (and society at large) start relying on these mechanisms it would become much harder to survive as a fugitive or dissident. furthermore, there comes another question: which government speficially would administer the system? even supposedly worldwide bodies like the united nations are not universally trusted, often precisely because they are such perfect targets for corruption among anyone trying to secure any kind of worldwide control. thus, to both avoid a corporation subverting the system for profit and a government subverting the system for its own political ends, placing the power into the hands of a decentralized network, if possible, is arguably the best option. but how is it possible? identity corp can certainly avoid the truly difficult challenge of actively interacting with the world because all it does is provide information. however, receiving data about the world, including its users’ biometric information, would be nevertheless very challenging. there are no public apis for such information; the only option would be for some human agent, or group of agents, to collect it. the channel of communication between the humans and the network will be simply digital bits, so it is very easy to see how these agents themselves could defraud the system: they could create many different identities for fake individuals with fake data. the only solution seems to be, once again, decentralization and redundancy: have many different agents collecting the same information, and require individuals looking to get an identity to confirm it with several different agents, ideally randomly (or otherwise) selected by the system itself. these agents would all send out messages to the network containing both biometric data and the identity that data is mapped to, perhaps encrypted using some cryptographic mechanisms that allows two datasets to be checked to see if they are nearly identical but shows nothing else. if two different agents assign two biometric identities to the same data, the second identity can be rejected. if someone tries to register an identity with fake biometric data, they will need to convince a number of specific organizations to somehow accept it. finally, the system should also include a mechanism for detecting and correcting fraud after the fact, perhaps using some sort of special-purpose decentralized “court”. the second challenge is figuring out exactly who these “agents” are going to be. the system should be able to avoid sybil attacks (the technical term for an attacker pretending to be a million entities so as to take control of a network that relies on consensus), and weed out bad agents without that mechanism itself being subject to bad agents or sybil attacks. proof-of-work and proof-of-stake is not enough; since we do not want each individual to travel around the world giving their biometric information to 51% of the network, in practice it may only take as little as 10% or even 5% to pull off fraud on a large scale. thus, it is quite probable that making a pure decentralized corporation to accomplish this task will be impossible; rather, the best we can hope for is a hybrid system that uses heavy support from humans to keep the network in balance, but at the same time uses the network’s cryptographic properties to force the system to stick to its original mission. this would be somewhere between a legal contract or constitution and a true decentralized network, but the distinction there is a very fluid one; as lawrence lessig is keen to point out, “code is law“. socialcoin and the one world “government” the existence of a decentralized “real name” system allows for a large number of possibilities that have so far been unexplored in the cryptocurrency world. one attractive possibility is socialcoin, the cryptocurrency that pays everyone in the world a “world citizen’s dividend” of 1000 units per month; another, similar alternative is to plug the system into a devcoin-like system, allowing people to come together and vote on projects that the money should be spent on, thereby creating what is essentially a (voluntary) “world government” that funds itself from the revenue from generating new currency units. how much money could such a government get while still maintaining a low inflation rate? here, there are two factors to keep in mind: people dying and losing their coins forever, and actual inflation. currently, when someone dies, their property automatically goes to their children or spouse by default. in a cryptocurrency, however, by default a person’s monetary savings simply become inaccessible since their passwords are lost. this destruction of coins creates a deflationary pressure; given the current death rate of around 8 per 1000 per year, multiplying by a factor of 2 to account for the fact that people tend to be somewhat wealthier than average at the time of their death, and then again dividing by 3 to take into account the fact that many people will have a system set up to ensure their wealth will go somewhere when they die (currently, about half the population has wills, and the divider can be bumped to 3 since people with more money are more likely to have them), we can get an estimate of 0.5% coin loss per year. this, combined with a low target inflation rate of 1.5%, means that we can “print” 2% of the current money supply every year. since cryptocurrencies will massively reduce the amount of fractional reserve banking in the world (as the cryptocurrency base unit is online, so individuals no longer “need” to store their money in banks in order to maintain savings accounts and make long-distance transactions), we can expect much of the world’s m2 and m3 money supply (ways of calculating money supply that include bank deposits) to become part of the base money supply of a cryptocurrency. the m2 money supply of the world is estimated at around 40trillion</a>,givingourworldgovernmentabudgetof40 trillion</a>, giving our world government a budget of 40trillion</a>,givingourworldgovernmentabudgetof800 billion per year to play with – or, in the case of socialcoin, a universal dividend of $114 per person per year. in theory, a world government can do a lot with 800billionperyear;inpractice,itremainstobeseenhowfreefromcorruptionsuchaninstitutionwouldbe,althoughinthiscasethefactthatitwillbecontrolledbydirectdemocracy,andhavenopowertotax,canpotentiallyserveaspowerfulrestraintsonabuse.itwouldessentiallybeagovernmentinthesenseofbeinganentitytaskedwithmaintainingsocialinfrastructure,butwouldlackthepowertocoerceandcompelthatmightmakeitparticularlydangerous.or,wecansimplystickwithsocialcoin,andleaveituptoeachindividualtoimprovetheirlivesthebestthattheycanwith800 billion per year; in practice, it remains to be seen how free from corruption such an institution would be, although in this case the fact that it will be controlled by direct democracy, and have no power to tax, can potentially serve as powerful restraints on abuse. it would essentially be a government in the sense of being an entity tasked with maintaining social infrastructure, but would lack the power to coerce and compel that might make it particularly dangerous. or, we can simply stick with socialcoin, and leave it up to each individual to improve their lives the best that they can with 800billionperyear;inpractice,itremainstobeseenhowfreefromcorruptionsuchaninstitutionwouldbe,althoughinthiscasethefactthatitwillbecontrolledbydirectdemocracy,andhavenopowertotax,canpotentiallyserveaspowerfulrestraintsonabuse.itwouldessentiallybeagovernmentinthesenseofbeinganentitytaskedwithmaintainingsocialinfrastructure,butwouldlackthepowertocoerceandcompelthatmightmakeitparticularlydangerous.or,wecansimplystickwithsocialcoin,andleaveituptoeachindividualtoimprovetheirlivesthebestthattheycanwith114 per year – almost nothing to most people reading this article, but a very substantial amount in many underdeveloped countries. if the system can be made to rely on no centralized institutions and no tax revenue, it can secure a level of political neutrality that would allow it to be trusted by the entire world. will it happen? well, either wait and see to find out, or start implementing it yourself. see also: http://bitcoinmagazine.com/7050/bootstrapping-a-decentralized-autonomous-corporation-part-i/ http://bitcoinmagazine.com/7119/bootstrapping-an-autonomous-decentralized-corporation-part-2-interacting-with-the-world/ previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements scalability, part 3: on metacoin history and multichain | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search scalability, part 3: on metacoin history and multichain posted by vitalik buterin on november 13, 2014 research & development special thanks to vlad zamfir for much of the thinking behind multi-chain cryptoeconomic paradigms first off, a history lesson. in october 2013, when i was visiting israel as part of my trip around the bitcoin world, i came to know the core teams behind the colored coins and mastercoin projects. once i properly understood mastercoin and its potential, i was immediately drawn in by the sheer power of the protocol; however, i disliked the fact that the protocol was designed as a disparate ensemble of "features", providing a subtantial amount of functionality for people to use, but offering no freedom to escape out of that box. seeking to improve mastercoin's potential, i came up with a draft proposal for something called "ultimate scripting" a general-purpose stack-based programming language that mastercoin could include to allow two parties to make a contract on an arbitrary mathematical formula. the scheme would generalize savings wallets, contracts for difference, many kinds of gambling, among other features. it was still quite limited, allowing only three stages (open, fill, resolve) and no internal memory and being limited to 2 parties per contract, but it was the first true seed of the ethereum idea. i submitted the proposal to the mastercoin team. they were impressed, but elected not to adopt it too quickly out of a desire to be slow and conservative; a philosophy which the project keeps to to this day and which david johnston mentioned at the recent tel aviv conference as mastercoin's primary differentiating feature. thus, i decided to go out on my own and simply build the thing myself. over the next three weeks i created the original ethereum whitepaper (unfortunately now gone, but a still very early version exists here). the basic building blocks were all there, except the progamming language was register-based instead of stack-based, and, because i was/am not skilled enough in p2p networking to build an independent blockchain client from scratch, it was to be built as a meta-protocol on top of primecoin not bitcoin, because i wanted to satisfy the concerns of bitcoin developers who were angry at meta-protocols bloating the blockchain with extra data. once competent developers like gavin wood and jeffrey wilcke, who did not share my deficiencies in ability to write p2p networking code, joined the project, and once enough people were excited that i saw there would be money to hire more, i made the decision to immediately move to an independent blockchain. the reasoning for this choice i described in my whitepaper in early january: the advantage of a metacoin protocol is that it can allow for more advanced transaction types, including custom currencies, decentralized exchange, derivatives, etc, that are impossible on top of bitcoin itself. however, metacoins on top of bitcoin have one major flaw: simplified payment verification, already difficult with colored coins, is outright impossible on a metacoin. the reason is that while one can use spv to determine that there is a transaction sending 30 metacoins to address x, that by itself does not mean that address x has 30 metacoins; what if the sender of the transaction did not have 30 metacoins to start with and so the transaction is invalid? finding out any part of the current state essentially requires scanning through all transactions going back to the metacoin's original launch to figure out which transactions are valid and which ones are not. this makes it impossible to have a truly secure client without downloading the entire 12 gb bitcoin blockchain. essentially, metacoins don't work for light clients, making them rather insecure for smartphones, users with old computers, internet-of-things devices, and once the blockchain scales enough for desktop users as well. ethereum's independent blockchain, on the other hand, is specifically designed with a highly advanced light client protocol; unlike with meta-protocols, contracts on top of ethereum inherit the ethereum blockchain's light client-friendliness properties fully. finally, long after that, i realized that by making an independent blockchain allows us to experiment with stronger versions of ghost-style protocols, safely knocking down the block time to 12 seconds. so what's the point of this story? essentially, had history been different, we easily could have gone the route of being "on top of bitcoin" right from day one (in fact, we still could make that pivot if desired), but solid technical reasons existed then why we deemed it better to build an independent blockchain, and these reasons still exist, in pretty much exactly the same form, today. since a number of readers were expecting a response to how ethereum as an independent blockchain would be useful even in the face of the recent announcement of a metacoin based on ethereum technology, this is it. scalability. if you use a metacoin on btc, you gain the benefit of having easier back-and-forth interaction with the bitcoin blockchain, but if you create an independent chain then you have the ability to achieve much stronger guarantees of security particularly for weak devices. there are certainly applications for which a higher degree of connectivity with btc is important ; for these cases a metacoin would certainly be superior (although note that even an independent blockchain can interact with btc pretty well using basically the same technology that we'll describe in the rest of this blog post). thus, on the whole, it will certainly help the ecosystem if the same standardized evm is available across all platforms. beyond 1.0 however, in the long term, even light clients are an ugly solution. if we truly expect cryptoeconomic platforms to become a base layer for a very large amount of global infrastructure, then there may well end up being so many crypto-transactions altogether that no computer, except maybe a few very large server farms run by the likes of google and amazon, is powerful enough to process all of them. thus, we need to break the fundamental barrier of cryptocurrency: that there need to exist nodes that process every transaction. breaking that barrier is what gets a cryptoeconomic platform's database from being merely massively replicated to being truly distributed. however, breaking the barrier is hard, particularly if you still want to maintain the requirement that all of the different parts of the ecosystem should reinforce each other's security. to achieve the goal, there are three major strategies: building protocols on top of ethereum that use ethereum only as an auditing-backend-of-last-resort, conserving transaction fees. turning the blockchain into something much closer to a high-dimensional interlinking mesh with all parts of the database reinforcing each other over time. going back to a model of one-protocol (or one service)-per-chain, and coming up with mechanisms for the chains to (1) interact, and (2) share consensus strength. of these strategies, note that only (1) is ultimately compatible with keeping the blockchain in a form anything close to what the bitcoin and ethereum protocols support today. (2) requires a massive redesign of the fundamental infrastructure, and (3) requires the creation of thousands of chains, and for fragility mitigation purposes the optimal approach will be to use thousands of currencies (to reduce the complexity on the user side, we can use stable-coins to essentially create a common cross-chain currency standard, and any slight swings in the stable-coins on the user side would be interpreted in the ui as interest or demurrage so the user only needs to keep track of one unit of account). we already discussed (1) and (2) in previous blog posts, and so today we will provide an introduction to some of the principles involved in (3). multichain the model here is in many ways similar to the bitshares model, except that we do not assume that dpos (or any other pos) will be secure for arbitrarily small chains. rather, seeing the general strong parallels between cryptoeconomics and institutions in wider society, particularly legal systems, we note that there exists a large body of shareholder law protecting minority stakeholders in real-world companies against the equivalent of a 51% attack (namely, 51% of shareholders voting to pay 100% of funds to themselves), and so we try to replicate the same system here by having every chain, to some degree, "police" every other chain either directly or indirectly through an interlinking transitive graph. the kind of policing required is simple policing aganist double-spends and censorship attacks from local majority coalitions, and so the relevant guard mechanisms can be implemented entirely in code. however, before we get to the hard problem of inter-chain security, let us first discuss what actually turns out to be a much easier problem: inter-chain interaction. what do we mean by multiple chains "interacting"? formally, the phrase can mean one of two things: internal entities (ie. scripts, contracts) in chain a are able to securely learn facts about the state of chain b (information transfer) it is possible to create a pair of transactions, t in a and t' in b, such that either both t and t' get confirmed or neither do (atomic transactions) a sufficiently general implementation of (1) implies (2), since "t' was (or was not) confirmed in b" is a fact about the state of chain b. the simplest way to do this is via merkle trees, described in more detail here and here; essentially merkle trees allow the entire state of a blockchain to be hashed into the block header in such a way that one can come up with a "proof" that a particular value is at a particular position in the tree that is only logarithmic in size in the entire state (ie. at most a few kilobytes long). the general idea is that contracts in one chain validate these merkle tree proofs of contracts in the other chain. a challenge that is greater for some consensus algorithms than others is, how does the contract in a chain validate the actual blocks in another chain? essentially, what you end up having is a contract acting as a fully-fledged "light client" for the other chain, processing blocks in that chain and probabilistically verifying transactions (and keeping track of challenges) to ensure security. for this mechanism to be viable, at least some quantity of proof of work must exist on each block, so that it is not possible to cheaply produce many blocks for which it is hard to determine that they are invalid; as a general rule, the work required by the blockmaker to produce a block should exceed the cost to the entire network combined of rejecting it. additionally, we should note that contracts are stupid; they are not capable of looking at reputation, social consensus or any other such "fuzzy" metrics of whether or not a given blockchain is valid; hence, purely "subjective" ripple-style consensus will be difficult to make work in a multi-chain setting. bitcoin's proof of work is (fully in theory, mostly in practice) "objective": there is a precise definition of what the current state is (namely, the state reached by processing the chain with the longest proof of work), and any node in the world, seeing the collection of all available blocks, will come to the same conclusion on which chain (and therefore which state) is correct. proof-of-stake systems, contrary to what many cryptocurrency developers think, can be secure, but need to be "weakly subjective" that is, nodes that were online at least once every n days since the chain's inception will necessarily converge on the same conclusion, but long-dormant nodes and new nodes need a hash as an initial pointer. this is needed to prevent certain classes of unavoidable long-range attacks. weakly subjective consensus works fine with contracts-as-automated-light-clients, since contracts are always "online". note that it is possible to support atomic transactions without information transfer; tiernolan's secret revelation protocol can be used to do this even between relatively dumb chains like btc and doge. hence, in general interaction is not too difficult. security the larger problem, however, is security. blockchains are vulnerable to 51% attacks, and smaller blockchains are vulnerable to smaller 51% attacks. ideally, if we want security, we would like for multiple chains to be able to piggyback on each other's security, so that no chain can be attacked unless every chain is attacked at the same time. within this framework, there are two major paradigm choices that we can make: centralized or decentralized. centralized decentralized a centralized paradigm is essentially every chain, whether directly or indirectly, piggybacking off of a single master chain; bitcoin proponents often love to see the central chain being bitcoin, though unfortunately it may be something else since bitcoin was not exactly designed with the required level of general-purpose functionality in mind. a decentralized paradigm is one that looks vaguely like ripple's network of unique node lists, except working across chains: every chain has a list of other consensus mechanisms that it trusts, and those mechanisms together determine block validity. the centralized paradigm has the benefit that it's simpler; the decentralized paradigm has the benefit that it allows for a cryptoeconomy to more easily swap out different pieces for each other, so it does not end up resting on decades of outdated protocols. however, the question is, how do we actually "piggyback" on one or more other chains' security? to provide an answer to this question, we'll first come up with a formalism called an assisted scoring function. in general, the way blockchains work is they have some scoring function for blocks, and the top-scoring block becomes the block defining the current state. assisted scoring functions work by scoring blocks based on not just the blocks themselves, but also checkpoints in some other chain (or multiple chains). the general principle is that we use the checkpoints to determine that a given fork, even though it may appear to be dominant from the point of view of the local chain, can be determined to have come later through the checkpointing process. a simple approach is that a node penalizes forks where the blocks are too far apart from each other in time, where the time of a block is determined by the median of the earliest known checkpoint of that block in the other chains; this would detect and penalize forks that happen after the fact. however, there are two problems with this approach: an attacker can submit the hashes of the blocks into the checkpoint chains on time, and then only reveal the blocks later an attacker may simply let two forks of a blockchain grow roughly evenly simultaneously, and then eventually push on his preferred fork with full force to deal with (2), we can say that only the valid block of a given block number with the earliest average checkpointing time can be part of the main chain, thus essentially completely preventing double-spends or even censorship forks; every new block would have to point to the last known previous block. however, this does nothing against (1). to solve (1), the best general solutions involve some concept of "voting on data availability" (see also: jasper den ouden's previous post talking about a similar idea); essentially, the participants in the checkpointing contract on each of the other chains would schelling-vote on whether or not the entire data of the block was available at the time the checkpoint was made, and a checkpoint would be rejected if the vote leans toward "no". for a block to be valid, it must be signed off on by a positive result from one or more external schelling-vote mechanisms note that there are two versions of this strategy. the first is a strategy where participants vote on data availability only (ie. that every part of the block is out there online). this allows the voters to be rather stupid, and be able to vote on availability for any blockchain; the process for determining data availability simply consists of repeatedly doing a reverse hash lookup query on the network until all the "leaf nodes" are found and making sure that nothing is missing. a clever way to force nodes to not be lazy when doing this check is to ask them to recompute and vote on the root hash of the block using a different hash function. once all the data is available, if the block is invalid an efficient merkle-tree proof of invalidity can be submitted to the contract (or simply published and left for nodes to download when determining whether or not to count the given checkpoint). the second strategy is less modular: have the schelling-vote participants vote on block validity. this would make the process somewhat simpler, but at the cost of making it more chain-specific: you would need to have the source code for a given blockchain in order to be able to vote on it. thus, you would get fewer voters providing security for your chain automatically. regardless of which of these two strategies is used, the chain could subsidize the schelling-vote contract on the other chain(s) via a cross-chain exchange. the scalability part up until now, we still don't have any actual "scalability"; a chain is only as secure as the number of nodes that are willing to download (although not process) every block. of course, there are solutions to this problem: challenge-response protocols and randomly selected juries, both described in the previous blog post on hypercubes, are the two that are currently best-known. however, the solution here is somewhat different: instead of setting in stone and institutionalizing one particular algorithm, we are simply going to let the market decide. the "market" is defined as follows: chains want to be secure, and want to save on resources. chains need to select one or more schelling-vote contracts (or other mechanisms potentially) to serve as sources of security (demand) schelling-vote contracts serve as sources of security (supply). schelling-vote contracts differ on how much they need to be subsidized in order to secure a given level of participation (price) and how difficult it is for an attacker to bribe or take over the schelling-vote to force it to deliver an incorrect result (quality). hence, the cryptoeconomy will naturally gravitate toward schelling-vote contracts that provide better security at a lower price, and the users of those contracts will benefit from being afforded more voting opportunities. however, simply saying that an incentive exists is not enough; a rather large incentive exists to cure aging and we're still pretty far from that. we also need to show that scalability is actually possible. the better of the two algorithms described in the post on hypercubes, jury selection, is simple. for every block, a random 200 nodes are selected to vote on it. the set of 200 is almost as secure as the entire set of voters, since the specific 200 are not picked ahead of time and an attacker would need to control over 40% of the participants in order to have any significant chance of getting 50% of any set of 200. if we are separating voting on data availability from voting on validity, then these 200 can be chosen from the set of all participants in a single abstract schelling-voting contract on the chain, since it's possible to vote on the data availability of a block without actually understanding anything about the blockchain's rules. thus, instead of every node in the network validating the block, only 200 validate the data, and then only a few nodes need to look for actual errors, since if even one node finds an error it will be able to construct a proof and warn everyone else. conclusion so, what is the end result of all this? essentially, we have thousands of chains, some with one application, but also with general-purpose chains like ethereum because some applications benefit from the extremely tight interoperability that being inside a single virtual machine offers. each chain would outsource the key part of consensus to one or more voting mechanisms on other chains, and these mechanisms would be organized in different ways to make sure they're as incorruptible as possible. because security can be taken from all chains, a large portion of the stake in the entire cryptoeconomy would be used to protect every chain. it may prove necessary to sacrifice security to some extent; if an attacker has 26% of the stake then the attacker can do a 51% takeover of 51% of the subcontracted voting mechanisms or schelling-pools out there; however, 26% of stake is still a large security margin to have in a hypothetical multi-trillion-dollar cryptoeconomy, and so the tradeoff may be worth it. the true benefit of this kind of scheme is just how little needs to be standardized. each chain, upon creation, can choose some number of schelling-voting pools to trust and subsidize for security, and via a customized contract it can adjust to any interface. merkle trees will need to be compatible with all of the different voting pools, but the only thing that needs to be standardized there is the hash algorithm. different chains can use different currencies, using stable-coins to provide a reasonably consistent cross-chain unit of value (and, of course, these stable-coins can themselves interact with other chains that implement various kinds of endogenous and exogenous estimators). ultimately, the vision of one of thousands of chains, with the different chains "buying services" from each other. services might include data availability checking, timestamping, general information provision (eg. price feeds, estimators), private data storage (potentially even consensus on private data via secret sharing), and much more. the ultimate distributed crypto-economy. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle [mirror] central planning as overfitting 2018 nov 25 see all posts this is a mirror of the post at https://radicalxchange.org/blog/posts/2018-11-26-4m9b8b/ written by myself and glen weyl there is an intuition shared by many that "central planning" — command-and-control techniques for allocating resources in economies, and fine-grained knob-turning interventionism more generally — is undesirable. there's quite a lot to this, but it is often misapplied in a way that also leads it to go under-appreciated. in this post we try to clarify the appropriate scope of the intuition. some recent examples of the intuition being misapplied are: people arguing that relatively simple entitlement programs like social security are burdensome government intervention, while elaborate and often discretionary tax breaks conditional on specific behaviors are a good step towards less government. people arguing that block size limits in cryptocurrencies, which impose a hard cap on the number of transactions that each block can contain, are a form of central planning, but who do not argue against other centrally planned parameters, eg. the targeted ten minute (or whatever) average time interval between blocks. people arguing that a lack of block size limits constitutes central planning (!!) people arguing that fixed transaction fees constitute central planning, but variable transaction fees that arise out of an equilibrium itself created by a fixed block size limit do not. we were recently at a discussion in policy circles in washington, where one of us was arguing for a scheme based on harberger taxes for spectrum licenses, debating against someone defending more conventional perpetual monopoly licenses on spectrum aggregated at large levels that would tend to create a few national cellular carriers. the latter side argued that the harberger tax scheme constituted unacceptable bureaucratic interventionism, but seemed to believe that permanent government-issued monopoly privileges are a property right as natural as apples falling from a tree. while we do not entirely dismiss this last example, for reasons we will return to later, it does seem overplayed. similarly and conversely, we see many examples where, in the name of defending or promoting "markets" (or at least "economic rationality") many professional economists advocate schemes that seem much more to us like central planning than the systems they seek to replace: the most systematic example of this is the literature on "optimal mechanism design," which began with the already extremely complicated and fragile vickrey-clarke-groves mechanism and has only gotten more byzantine from there. while vickrey's motivation for these ideas was to discover relatively simple rules that would correct the flaws of standard capitalism, he acknowledged in his paper that the design was highly complex in its direct application and urged future researchers to find simplifications. instead of following this counsel, many scholars have proposed, for example, schemes that rely on a central authority being able to specify an infinite dimensional set of prior beliefs. these schemes, we submit, constitute "central planning" in precisely the sense we should be concerned with. furthermore, these designs are not just matters of theory, but in practice many applied mechanism designers have created systems with similar properties. the recent united states spectrum incentive auctions (designed by a few prominent economists and computer scientists) centralized the enforcement of potential conflicts between transmission rights using an extremely elaborate and opaque computational engine, rather than allowing conflicts to be resolved through (for example) common law liability lawsuits as other interference between property claims and land uses are. a recent design for the allocation of courses to students at the university of pennsylvania designed by a similar team requires students to express their preferences over courses on a novel numerical scale, allowing them only narrow language for expressing complementarities and substitutability between courses and then uses a state-of-the-art optimization engine to allocate courses. auction systems designed by economists and computer scientists at large technology companies, like facebook and google, are even richer and less transparent, and have created substantial backlash, inspiring a whole industry of firms that help advertisers optimize their bidding in elaborate ways against these systems. this problem does not merely arise in mechanism design, however. in the fields of industrial organization (the basis of much antitrust economics) and the field of macroeconomics (the basis of much monetary policy), extremely elaborate models with hundreds of parameters are empirically estimated and used to simulate the effects of mergers or changes in monetary policy. these models are usually difficult to explain even to experts in the field, much less democratically-elected politicians, judges or, god forbid, voters. and yet the confidence we have in these models, the empirical evidence validating their accuracy, etc. is almost nothing. nonetheless, economists consistently promote such methods as "the state of the art" and they are generally viewed positively by defenders of the "market economy". to understand why we think the concept of "intervention" is being misapplied here, we need to understand two different ways of measuring the extent to which some scheme is "interventionist". the first approach is to try to measure the absolute magnitude of distortion relative to some imagined state of nature (anarcho-primitivism, or a blockchain with no block size limits, or...). however, this approach clearly fails to capture the intuitions of why central planning is undesirable. for example, property rights in the physical world are a large intervention into almost every person's behavior, considerably limiting the actions that we can take every day. many of these restrictions are actually of quite recent historical provenance (beginning with agriculture, and mostly in the west and not the east or middle east). however, opponents of central planning often tend to be the strongest proponents of property rights! we can shed some light on this puzzle by looking at another way of measuring the "central-planny-ness" of some social structure: in short, measure the number of knobs. property rights actually score quite well here: every piece of property is allocated to some person or legal entity, they can use it as they wish, and no one else can touch it without their permission. there are choices to make around the edges (eg. adverse possession rights), but generally there isn't too much room for changing the scheme around (though note that privatization schemes, ie. transitions from something other than property rights to property rights like the auctions we discussed above, have very many knobs, and so there we can see more risks). command-and-control regulations with ten thousand clauses (or market designs that specify elaborate probabilistic objects, or optimization protocols, etc.), or attempts to limit use of specific features of the blockchain to drive out specific categories of users, are much less desirable, as such strategies leave many more choices to central planners. a block size limit and a fixed transaction fee (or carbon taxes and a cap-and-trade scheme) have the exact same level of "central-planny-ness" to them: one variable (either quantity or price) is fixed in the protocol, and the other variable is left to the market. here are some key underlying reasons why we believe that simple social systems with fewer knobs are so desirable: they have fewer moving parts that can fail or otherwise have unexpected effects. they are less likely to overfit. if a social system is too complex, there are many parameters to set and relatively few experiences to draw from when setting the parameters, making it more likely that the parameters will be set to overfit to one context, and generalize poorly to other places or to future times. we know little, and we should not design systems that demand us to know a lot. they are more resistant to corruption. if a social system is simpler, it is more difficult (not impossible, but still more difficult) to set parameters in ways that encode attempts to privilege or anti-privilege specific individuals, factions or communities. this is not only good because it leads to fairness, it is also good because it leads to less zero-sum conflict. they can more easily achieve legitimacy. because simpler systems are easier to understand, and easier for people to understand that a given implementation is not unfairly privileging special interests, it is easier to create common knowledge that the system is fair, creating a stronger sense of trust. legitimacy is perhaps the central virtue of social institutions, as it sustains cooperation and coordination, enables the possibility of democracy (how can you democratically participate and endorse a system you do not understand?) and allows for a bottoms-up, rather than top-down, creation of a such a system, ensuring it can be implemented without much coercion or violence. these effects are not always achieved (for example, even if a system has very few knobs, it's often the case that there exists a knob that can be turned to privilege well-connected and wealthy people as a class over everyone else), but the simpler a system is, the more likely the effects are to be achieved. while avoiding over-complexity and overfit in personal decision-making is also important, avoiding these issues in large-scale social systems is even more important, because of the inevitable possibility of powerful forces attempting to manipulate knobs for the benefit of special interests, and the need to achieve common knowledge that the system has not been greatly corrupted, to the point where the fairness of the system is obvious even to unsophisticated observers. this is not to condemn all forms or uses of complexity in social systems. most science and the inner workings of many technical systems are likely to be opaque to the public but this does not mean science or technology is useless in social life; far from it. however, these systems, to gain legitimacy, usually show that they can reliably achieve some goal, which is transparent and verifiable. planes land safely and on time, computational systems seem to deliver calculations that are correct, etc. it is by this process of verification, rather than by the transparency of the system per se, that such systems gain their legitimacy. however, for many social systems, truly large-scale, repeatable tests are difficult if not impossible. as such, simplicity is usually critical to legitimacy. different notions of simplicity however, there is one class of social systems that seem to be desirable, and that intellectual advocates of minimizing central planning tend to agree are desirable, that don't quite fit the simple "few knobs" characterization that we made above. for example, consider common law. common law is built up over thousands of precedents, and contains a large number of concepts (eg. see this list under "property law", itself only a part of common law; have you heard of "quicquid plantatur solo, solo cedit" before?). however, proponents of private property are very frequently proponents of common law. so what gives? here, we need to make a distinction between redundant complexity, or many knobs that really all serve a relatively small number of similar goals, and optimizing complexity, in the extreme one knob per problem that the system has encountered. in computational complexity theory, we typically talk about kolmogorov complexity, but there are other notions of complexity that are also useful here, particularly vc dimension roughly, the size of the largest set of situations for which we can turn the knobs in a particular way to achieve any particular set of outcomes. many successful machine learning techniques, such as support vector machines and boosting, are quite complex, both in the formal kolmogorov sense and in terms of the outcomes they produce, but can be proven to have low vc dimension. vc dimension does a nice job capturing some of the arguments for simplicity mentioned above more explicitly, for example: a system with low vc dimension may have some moving parts that fail, but if it does, its different constituent parts can correct for each other. by construction, it has built in resilience through redundancy low vc dimension is literally a measure of resistance to overfit. low vc dimension leads to resistance to corruption, because if vc dimension is low, a corrupt or self-interested party in control of some knobs will not as easily be able to achieve some particular outcome that they desire. in particular, this agent will be "checked and balanced" by other parts of the system that redundantly achieve the originally desired ends. they can achieve legitimacy because people can randomly check a few parts and verify in detail that those parts work in ways that are reasonable, and assume that the rest of the system works in a similar way. an example of this was the ratification of the united states constitution which, while quite elaborate, was primarily elaborate in the redundancy with which it applied the principle of checks and balances of power. thus most citizens only read one or a few of the federalist papers that explained and defended the constitution, and yet got a reasonable sense for what was going on. this is not as clean and convenient as a system with low kolmogorov complexity, but still much better than a system with high complexity where the complexity is "optimizing" (for an example of this in the blockchain context, see vitalik's opposition and alternative to on-chain governance). the primary disadvantage we see in kolmogorov complex but vc simple designs is for new social institutions, where it may be hard to persuade the public that these are vc simple. vc simplicity is usually easier as a basis for legitimacy when an institutions has clearly been built up without any clear design over a long period of time or by a large committee of people with conflicting interests (as with the united states constitution). thus when offering innovations we tend to focus more on kolmogorov simplicity and hope many redundant each kolmogorov-simple elements will add up to a vc-simple system. however, we may just not have the imagination to think of how vc simplicity might be effectively explained. there are forms of the "avoid central planning" intuition that are misunderstandings and ultimately counterproductive. for example, try to automatically seize upon designs that seem at first glance to "look like a market", because not all markets are created equal. for example, one of us has argued for using fixed prices in certain settings to reduce uncertainty, and the other has (for similar information sharing reasons) argued for auctions that are a synthesis of standard descending price dutch and ascending price english auctions (channel auctions). that said, it is also equally a large error to throw the intuition away entirely. rather, it is a valuable and important insight that can easily is central to the methodology we have been recently trying to develop. simplicity to whom? or why humanities matter however, the academic critics of this type of work are not simply confused. there is a reasonable basis for unease with discussions of "simplicity" because they inevitably contain a degree of subjectivity. what is "simple" to describe or appears to have few knobs in one language for describing it is devilishly complex in another, and vice versa. a few examples should help illuminate the point: we have repeatedly referred to "knobs", which are roughly real valued parameters. but real-valued parameters can encode an arbitrary amount of complexity. for example, i could claim my system has only one knob, it is just that slight changes in the 1000th decimal place of the setting of that knob end up determining incredibly important properties of the system. this may seem cheap, but more broadly it is the case that non-linear mappings between systems can make one system seem "simple" and another "complex" and in general there is just no way to say which is right. many think of the electoral system of the united states as "simple", and yet, if one reflects on it or tries to explain it to a foreigner, it is almost impossible to describe. it is familiar, not simple, and we just have given a label to it ("the american system") that lets us refer to it in a few words. systems like quadratic voting, or ranked choice voting, are often described as complex, but this seems to have more to do with lack of familiarity than complexity. many scientific concepts, such as the "light cone", are the simplest thing possible once one understands special relativity and yet are utterly foreign and bizarre without having wrapped one's hands around this theory. even kolmogorov complexity (length of the shortest computer program that encodes some given system) is relative to some programming language. now, to some extent, vc dimension offers a solution: it says that a class of systems is simple if it is not too flexible. but consider what happens when you try to apply this; to do so, let's return to our example upfront about harberger taxes v. perpetual licenses for spectrum. harberger taxes strike us as quite simple: there is a single tax rate (and the theory even says this is tied down by the rate at which assets turnover, at least if we want to maximally favor allocative efficiency) and the system can be described in a sentence or two. it seems pretty clear that such a system could not be contorted to achieve arbitrary ends. however, an opponent could claim that we chose the harberger tax from an array of millions of possible mechanisms of a similar class to achieve a specific objective, and it just sounds simple (as with our examples of "deceptive" simplicity above). to counter this argument, we would respond that the harberger tax, or very similar ideas, have been repeatedly discovered or used (to some success) throughout human history, beginning with the greeks, and that we do not propose this system simply for spectrum licenses but in a wide range of contexts. the chances that in all these contexts we are cherry-picking the system to "fit" that setting seems low. we would submit to the critic to judge whether it is really plausible that all these historical circumstances and these wide range of applications just "happen" to coincide. focusing on familiarity (ie. conservatism), rather than simplicity in some abstract mathematical sense, also carries many of the benefits of simplicity as we described above; after all, familiarity is simplicity, if the language we are using to describe ideas includes references to our shared historical experience. familiar mechanisms also have the benefit that we have more knowledge of how similar ideas historically worked in practice. so why not just be conservative, and favor perpetual property licenses strongly over harberger taxes? there are three flaws in that logic, it seems to us. first, to the extent it is applied, it should be applied uniformly to all innovation, not merely to new social institutions. technologies like the internet have contributed greatly to human progress, but have also led to significant social upheavals; this is not a reason to stop trying to advance our technologies and systems for communication, and it is not a reason to stop trying to advance our social technologies for allocating scarce resources. second, the benefits of innovation are real, and social institutions stand to benefit from growing human intellectual progress as much as everything else. the theoretical case for harberger taxes providing efficiency benefits is strong, and there is great social value in doing small and medium-scale experiments to try ideas like them out. investing in experiments today increases what we know, and so increases the scope of what can be done "conservatively" tomorrow. third, and most importantly, the cultural context in which you as a decision maker have grown up today is far from the only culture that has existed on earth. even at present, singapore, china, taiwan and scandinavia have had significant success with quite different property regimes than the united states. video game developers and internet protocol designers have had to solve incentive problems of a similar character to what we see today in the blockchain space and have come up with many kinds of solutions, and throughout history, we have seen a wide variety of social systems used for different purposes, with a wide range of resulting outcomes. by learning about the different ways in which societies have lived, understood what is natural and imagined their politics, we can gain the benefits of learning from historical experience and yet at the same time open ourselves to a much broader space of possible ideas to work with. this is why we believe that balance and collaboration between different modes of learning and understanding, both the mathematical one of economists and computer scientists, and the historical experiences studied by historians, anthropologists, political scientists, etc is critical to avoid the mix of and often veering between extreme conservatism and dangerous utopianism that has become characteristic of much intellectual discourse in e.g. the economics community, the "rationalist" community, and in many cases blockchain protocol design. a call to all the bug bounty hunters out there… | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search a call to all the bug bounty hunters out there… posted by jutta steiner on december 18, 2014 research & development hi, i’m jutta! as some of you might have read in earlier posts, i’ve recently been busy setting up a security audit prior to the ethereum genesis block release. ethereum will launch following a world-class review by experts in it security, cryptography and blockchain technology. prior to the launch, we will also complete a bug bounty program a major cornerstone of our approach to achieving security. the bug bounty program will rely on the ethereum community and all other motivated bug bounty hunters out there. we’ll soon release the final details of the program, currently under development by gustav. a first glimpse: 11 figure total satoshi rewards. leaderboard listing top contributors by total hunting score. other l33t ideas for rewards soon to be announced… just wondering if a contract in our genesis block could be the perfect place to eternalize the hall of fame (: get ready for hunting down flaws in the protocols, go implementation and network code. for those looking for a distraction over the holidays, please note that the protocols and code-base are currently still subject to change. please also note that rewards will only be given to submissions received after the official launch of the bounty program. detailed submission guidelines, rules and other info on the exact scope will soon be published on http://ethdev.com. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements important statement regarding the ether pre-sale | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search important statement regarding the ether pre-sale posted by stephan tual on february 13, 2014 organizational the ethereum project has had the incredible privilege to launch its poc testnet and engage the crypto-currency community over the past two months. during our experiences, we’ve encountered a lot of passionate support and wonderful questions that have helped us refine our thoughts and goals including the process we will eventually use to sell ether. this said, we have not finalized the structure and format for the ether presale and thus we do not recommend, encourage, or endorse any attempt to sell, trade, or acquire ether. we have recently become aware of peercover.com announcing a fundraising structure based in some way upon etherthey are in no way associated with the ethereum project, do not speak for it, and are, in our opinion, doing a disservice to the ethereum community by possibly leading their own clients into a situation that they don’t understand. offering to sell ether that doesn’t yet exist to mislead purchasers can only be considered irresponsible at this point. buyer beware. we request that peercover.com cease to offer ether forwards, until there is more information released on the ethereum project, the potential value of the ether cryptofuel, and until lawyers in various countries clarify what the securities and regulatory issues might be in selling ether to the public in various countries. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle starks, part ii: thank goodness it's fri-day 2017 nov 22 see all posts special thanks to eli ben-sasson for ongoing help and explanations, and justin drake for reviewing in the last part of this series, we talked about how you can make some pretty interesting succinct proofs of computation, such as proving that you have computed the millionth fibonacci number, using a technique involving polynomial composition and division. however, it rested on one critical ingredient: the ability to prove that at least the great majority of a given large set of points are on the same low-degree polynomial. this problem, called "low-degree testing", is perhaps the single most complex part of the protocol. we'll start off by once again re-stating the problem. suppose that you have a set of points, and you claim that they are all on the same polynomial, with degree less than \(d\) (ie. \(deg < 2\) means they're on the same line, \(deg < 3\) means they're on the same line or parabola, etc). you want to create a succinct probabilistic proof that this is actually true. left: points all on the same \(deg < 3\) polynomial. right: points not on the same \(deg < 3\) polynomial if you want to verify that the points are all on the same degree \(< d\) polynomial, you would have to actually check every point, as if you fail to check even one point there is always some chance that that point will not be on the polynomial even if all the others are. but what you can do is probabilistically check that at least some fraction (eg. 90%) of all the points are on the same polynomial. top left: possibly close enough to a polynomial. top right: not close enough to a polynomial. bottom left: somewhat close to two polynomials, but not close enough to either one. bottom right: definitely not close enough to a polynomial. if you have the ability to look at every point on the polynomial, then the problem is easy. but what if you can only look at a few points that is, you can ask for whatever specific point you want, and the prover is obligated to give you the data for that point as part of the protocol, but the total number of queries is limited? then the question becomes, how many points do you need to check to be able to tell with some given degree of certainty? clearly, \(d\) points is not enough. \(d\) points are exactly what you need to uniquely define a degree \(< d\) polynomial, so any set of points that you receive will correspond to some degree \(< d\) polynomial. as we see in the figure above, however, \(d+1\) points or more do give some indication. the algorithm to check if a given set of values is on the same degree \(< d\) polynomial with \(d+1\) queries is not too complex. first, select a random subset of \(d\) points, and use something like lagrange interpolation (search for "lagrange interpolation" here for a more detailed description) to recover the unique degree \(< d\) polynomial that passes through all of them. then, randomly sample one more point, and check that it is on the same polynomial. note that this is only a proximity test, because there's always the possibility that most points are on the same low-degree polynomial, but a few are not, and the \(d+1\) sample missed those points entirely. however, we can derive the result that if less than 90% of the points are on the same degree \(< d\) polynomial, then the test will fail with high probability. specifically, if you make \(d+k\) queries, and if at least some portion \(p\) of the points are not on the same polynomial as the rest of the points, then the test will only pass with probability \((1 p)^k\). but what if, as in the examples from the previous article, \(d\) is very high, and you want to verify a polynomial's degree with less than \(d\) queries? this is, of course, impossible to do directly, because of the simple argument made above (namely, that any \(k \leq d\) points are all on at least one degree \(< d\) polynomial). however, it's quite possible to do this indirectly by providing auxiliary data, and achieve massive efficiency gains by doing so. and this is exactly what new protocols like fri ("fast rs iopp", rs = "reed-solomon", iopp = "interactive oracle proofs of proximity"), and similar earlier designs called probabilistically checkable proofs of proximity (pcpps), try to achieve. a first look at sublinearity to prove that this is at all possible, we'll start off with a relatively simple protocol, with fairly poor tradeoffs, but that still achieves the goal of sublinear verification complexity that is, you can prove proximity to a degree \(< d\) polynomial with less than \(d\) queries (and, for that matter, less than \(o(d)\) computation to verify the proof). the idea is as follows. suppose there are n points (we'll say \(n\) = 1 billion), and they are all on a degree \(<\) 1,000,000 polynomial \(f(x)\). we find a bivariate polynomial (ie. an expression like \(1 + x + xy + x^5 \cdot y^3 + x^{12} + x \cdot y^{11}\)), which we will denote \(g(x, y)\), such that \(g(x, x^{1000}) = f(x)\). this can be done as follows: for the \(k\)th degree term in \(f(x)\) (eg. \(1744 \cdot x^{185423}\)), we decompose it into \(x^{k \% 1000} \cdot y^{floor(k / 1000)}\) (in this case, \(1744 \cdot x^{423} \cdot y^{185}\)). you can see that if \(y = x^{1000}\), then \(1744 \cdot x^{423} \cdot y^{185}\) equals \(1744 \cdot x^{185423}\). in the first stage of the proof, the prover commits to (ie. makes a merkle tree of) the evaluation of \(g(x, y)\) over the entire square \([1 ... n] x \{x^{1000}: 1 \leq x \leq n\}\) that is, all 1 billion \(x\) coordinates for the columns, and all 1 billion corresponding thousandth powers for the \(y\) coordinates of the rows. the diagonal of the square represents the values of \(g(x, y)\) that are of the form \(g(x, x^{1000})\), and thus correspond to values of \(f(x)\). the verifier then randomly picks perhaps a few dozen rows and columns (possibly using the merkle root of the square as a source of pseudorandomness if we want a non-interactive proof), and for each row or column that it picks the verifier asks for a sample of, say, 1010 points on the row and column, making sure in each case that one of the points demanded is on the diagonal. the prover must reply back with those points, along with merkle branches proving that they are part of the original data committed to by the prover. the verifier checks that the merkle branches match up, and that the points that the prover provides actually do correspond to a degree-1000 polynomial. this gives the verifier a statistical proof that (i) most rows are populated mostly by points on degree \(< 1000\) polynomials, (ii) most columns are populated mostly by points on degree \(< 1000\) polynomials, and (iii) the diagonal line is mostly on these polynomials. this thus convinces the verifier that most points on the diagonal actually do correspond to a degree \(< 1,000,000\) polynomial. if we pick thirty rows and thirty columns, the verifier needs to access a total of 1010 points \(\cdot\) 60 rows+cols = 60600 points, less than the original 1,000,000, but not by that much. as far as computation time goes, interpolating the degree \(< 1000\) polynomials will have its own overhead, though since polynomial interpolation can be made subquadratic the algorithm as a whole is still sublinear to verify. the prover complexity is higher: the prover needs to calculate and commit to the entire \(n \cdot n\) rectangle, which is a total of \(10^{18}\) computational effort (actually a bit more because polynomial evaluation is still superlinear). in all of these algorithms, it will be the case that proving a computation is substantially more complex than just running it; but as we will see the overhead does not have to be that high. a modular math interlude before we go into our more complex protocols, we will need to take a bit of a digression into the world of modular arithmetic. usually, when we work with algebraic expressions and polynomials, we are working with regular numbers, and the arithmetic, using the operators +, -, \(\cdot\), / (and exponentiation, which is just repeated multiplication), is done in the usual way that we have all been taught since school: \(2 + 2 = 4\), \(72 / 5 = 14.4\), \(1001 \cdot 1001 = 1002001\), etc. however, what mathematicians have realized is that these ways of defining addition, multiplication, subtraction and division are not the only self-consistent ways of defining those operators. the simplest example of an alternate way to define these operators is modular arithmetic, defined as follows. the % operator means "take the remainder of": \(15 % 7 = 1\), \(53 % 10 = 3\), etc (note that the answer is always non-negative, so for example \(-1 % 10 = 9\)). for any specific prime number \(p\), we can redefine: \(x + y \rightarrow (x + y)\) % \(p\) \(x \cdot y \rightarrow (x \cdot y)\) % \(p\) \(x^y \rightarrow (x^y)\) % \(p\) \(x y \rightarrow (x y)\) % \(p\) \(x / y \rightarrow (x \cdot y ^{p-2})\) % \(p\) the above rules are all self-consistent. for example, if \(p = 7\), then: \(5 + 3 = 1\) (as \(8\) % \(7 = 1\)) \(1 3 = 5\) (as \(-2\) % \(7 = 5\)) \(2 \cdot 5 = 3\) \(3 / 5 = 2\) (as (\(3 \cdot 5^5\)) % \(7 = 9375\) % \(7 = 2\)) more complex identities such as the distributive law also hold: \((2 + 4) \cdot 3\) and \(2 \cdot 3 + 4 \cdot 3\) both evaluate to \(4\). even formulas like \((a^2 b^2)\) = \((a b) \cdot (a + b)\) are still true in this new kind of arithmetic. division is the hardest part; we can't use regular division because we want the values to always remain integers, and regular division often gives non-integer results (as in the case of \(3/5\)). the funny \(p-2\) exponent in the division formula above is a consequence of getting around this problem using fermat's little theorem, which states that for any nonzero \(x < p\), it holds that \(x^{p-1}\) % \(p = 1\). this implies that \(x^{p-2}\) gives a number which, if multiplied by \(x\) one more time, gives \(1\), and so we can say that \(x^{p-2}\) (which is an integer) equals \(\frac{1}{x}\). a somewhat more complicated but faster way to evaluate this modular division operator is the extended euclidean algorithm, implemented in python here. because of how the numbers "wrap around", modular arithmetic is sometimes called "clock math" with modular math we've created an entirely new system of arithmetic, and because it's self-consistent in all the same ways traditional arithmetic is self-consistent we can talk about all of the same kinds of structures over this field, including polynomials, that we talk about in "regular math". cryptographers love working in modular math (or, more generally, "finite fields") because there is a bound on the size of a number that can arise as a result of any modular math calculation no matter what you do, the values will not "escape" the set \(\{0, 1, 2 ... p-1\}\). fermat's little theorem also has another interesting consequence. if \(p-1\) is a multiple of some number \(k\), then the function \(x \rightarrow x^k\) has a small "image" that is, the function can only give \(\frac{p-1}{k} + 1\) possible results. for example, \(x \rightarrow x^2\) with \(p=17\) has only 9 possible results. with higher exponents the results are more striking: for example, \(x \rightarrow x^8\) with \(p=17\) has only 3 possible results. and of course, \(x \rightarrow x^{16}\) with \(p=17\) has only 2 possible results: for \(0\) it returns \(0\), and for everything else it returns \(1\). now a bit more efficiency let us now move on to a slightly more complicated version of the protocol, which has the modest goal of reducing the prover complexity from \(10^{18}\) to \(10^{15}\), and then \(10^{9}\). first, instead of operating over regular numbers, we are going to be checking proximity to polynomials as evaluated with modular math. as we saw in the previous article, we need to do this to prevent numbers in our starks from growing to 200,000 digits anyway. here, however, we are going to use the "small image" property of certain modular exponentiations as a side effect to make our protocols far more efficient. specifically, we will work with \(p =\) 1,000,005,001. we pick this modulus because (i) it's greater than 1 billion, and we need it to be at least 1 billion so we can check 1 billion points, (ii) it's prime, and (iii) \(p-1\) is an even multiple of 1000. the exponentiation \(x^{1000}\) will have an image of size 1,000,006 that is, the exponentiation can only give 1,000,006 possible results. this means that the "diagonal" (\(x\), \(x^{1000}\)) now becomes a diagonal with a wraparound; as \(x^{1000}\) can only take on 1,000,006 possible values, we only need 1,000,006 rows. and so, the full evaluation of \(g(x, x^{1000})\) now has only ~\(10^{15}\) elements. as it turns out, we can go further: we can have the prover only commit to the evaluation of \(g\) on a single column. the key trick is that the original data itself already contains 1000 points that are on any given row, so we can simply sample those, derive the degree \(< 1000\) polynomial that they are on, and then check that the corresponding point on the column is on the same polynomial. we then check that the column itself is \(a < 1000\) polynomial. the verifier complexity is still sublinear, but the prover complexity has now decreased to \(10^9\), making it linear in the number of queries (though it's still superlinear in practice because of polynomial evaluation overhead). and even more efficiency the prover complexity is now basically as low as it can be. but we can still knock the verifier complexity down further, from quadratic to logarithmic. and the way we do that is by making the algorithm recursive. we start off with the last protocol above, but instead of trying to embed a polynomial into a 2d polynomial where the degrees in \(x\) and \(y\) are equal, we embed the polynomial into a 2d polynomial where the degree bound in \(x\) is a small constant value; for simplicity, we can even say this must be 2. that is, we express \(f(x) = g(x, x^2)\), so that the row check always requires only checking 3 points on each row that we sample (2 from the diagonal plus one from the column). if the original polynomial has degree \(< n\), then the rows have degree \(< 2\) (ie. the rows are straight lines), and the column has degree \(< \frac{n}{2}\). hence, what we now have is a linear-time process for converting a problem of proving proximity to a polynomial of degree \(< n\) into a problem of proving proximity to a polynomial of degree \(< \frac{n}{2}\). furthermore, the number of points that need to be committed to, and thus the prover's computational complexity, goes down by a factor of 2 each time (eli ben-sasson likes to compare this aspect of fri to fast fourier transforms, with the key difference that unlike with ffts, each step of recursion only introduces one new sub-problem instead of branching out into two). hence, we can simply keep using the protocol on the column created in the previous round of the protocol, until the column becomes so small that we can simply check it directly; the total complexity is something like \(n + \frac{n}{2} + \frac{n}{4} + ... \approx 2n\). in reality, the protocol will need to be repeated several times, because there is still a significant probability that an attacker will cheat one round of the protocol. however, even still the proofs are not too large; the verification complexity is logarithmic in the degree, though it goes up to \(\log ^{2}n\) if you count the size of the merkle proofs. the "real" fri protocol also has some other modifications; for example, it uses a binary galois field (another weird kind of finite field; basically, the same thing as the 12th degree extension fields i talk about here, but with the prime modulus being 2). the exponent used for the row is also typically 4 and not 2. these modifications increase efficiency and make the system friendlier to building starks on top of it. however, these modifications are not essential to understanding how the algorithm works, and if you really wanted to, you could definitely make starks with the simple modular math-based fri described here too. soundness i will warn that calculating soundness that is, determining just how low the probability is that an optimally generated fake proof will pass the test for a given number of checks is still somewhat of a "here be dragons" area in this space. for the simple test where you take 1,000,000 \(+ k\) points, there is a simple lower bound: if a given dataset has the property that, for any polynomial, at least portion p of the dataset is not on the polynomial, then a test on that dataset will pass with at most \((1-p)^k\) probability. however, even that is a very pessimistic lower bound for example, it's not possible to be much more than 50% close to two low-degree polynomials at the same time, and the probability that the first points you select will be the one with the most points on it is quite low. for full-blown fri, there are also complexities involving various specific kinds of attacks. here is a recent article by ben-sasson et al describing soundness properties of fri in the context of the entire stark scheme. in general, the "good news" is that it seems likely that in order to pass the \(d(x) \cdot z(x) = c(p(x))\) check on the stark, the \(d(x)\) values for an invalid solution would need to be "worst case" in a certain sense they would need to be maximally far from any valid polynomial. this implies that we don't need to check for that much proximity. there are proven lower bounds, but these bounds would imply that an actual stark need to be ~1-3 megabytes in size; conjectured but not proven stronger bounds reduce the required number of checks by a factor of 4. the third part of this series will deal with the last major part of the challenge in building starks: how we actually construct constraint checking polynomials so that we can prove statements about arbitrary computation, and not just a few fibonacci numbers. ethereum: now going public | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum: now going public posted by vitalik buterin on january 23, 2014 research & development i first wrote the initial draft of the ethereum whitepaper on a cold day in san francisco in november, as a culmination of months of thought and often frustrating work into an area that we have come to call “cryptocurrency 2.0″ – in short, using the bitcoin blockchain for more than just money. in the months leading up to the development of ethereum, i had the privilege to work closely with several projects attempting to implement colored coins, smart property, and various types of decentralized exchange. at the time, i was excited by the sheer potential that these technologies could bring, as i was acutely aware that many of the major problems still plaguing the bitcoin ecosystem, including fraudulent services, unreliable exchanges, and an often surprising lack of security, were not caused by bitcoin’s unique property of decentralization; rather, these issues are a result of the fact that there was still great centralization left, in places where it could potentially quite easily be removed. what i soon realized, however, was the sheer difficulty that many of these projects were facing, and the often ugly technological hacks that were required to make them work. and, once one looks at the problem carefully, the culprit becomes obvious: fragmentation. each individual project was attempting to implement its own blockchain or meta-layer on top of bitcoin, and considerable effort was being duplicated and interoperability lost as a result. eventually, i realized that the key to solving the problem once and for all was a simple insight that the field of computer science first conceived in 1935: there is no need to construct a separate infrastructure for each individual feature and implementation; rather, it is possible to create a turing-complete programming language, and allow everyone to use that language to implement any feature that can be mathematically defined. this is how our computers work, and this is how our web browsers work; and, with ethereum, this is how our cryptocurrencies can work. since that moment, ethereum has come very far over the past two months. the ethereum team has expanded to include dozens of developers including gavin wood and jeffrey wilcke, lead developers of the c++ and go implementations, respectively, as well as others including charles hoskinson, anthony di iorio and mihai alisie, and dozens of other incredibly talented individuals who are unfortunately too many to mention. many of them have even come to understand the project so deeply as to be better at explaining ethereum than myself. there are now over fifteen people in our developer chat rooms actively working on the c++ and go implementations, which are already surprisingly close to having all the functionality needed to run in a testnet. aside from development effort, there are dozens of people operating around the world in our marketing and community outreach team, developing the non-technical infrastructure needed to make the ethereum ecosystem the solid and robust community that it deserves to be. and now, at this stage, we have made a collective decision that we are ready to take our organization much more public than we have been before. what is ethereum in short, ethereum is a next-generation cryptographic ledger that intends to support numerous advanced features, including user-issued currencies, smart contracts, decentralized exchange and even what we think is the first proper implementation of decentralized autonomous organizations (daos) or companies (dacs). however, this is not what makes ethereum special. rather, what makes ethereum special is the way that it does this. instead of attempting to specifically support each individual type of functionality as a feature, ethereum includes a built-in turing-complete scripting language, which allows you to code the features yourself through a mechanism known as “contracts”. a contract is like an autonomous agent that runs a certain piece of code every time a transaction is sent to it, and this code can modify the contract’s internal data storage or send transactions. advanced contracts can even modify their own code. a simple example of a contract would be a basic name registration system, allowing users to register their name with their address. this contract would not send transactions; its sole purpose is to build up a database which other nodes can then query. the contract, written in our high-level c-like language (cll) (or perhaps more accurately python-like language), looks as follows: if tx.value < block.basefee * 200: stop if contract.storage[tx.data[0]] or tx.data[0] < 100: stop contract.storage[tx.data[0]] = tx.data[1] and there you have it. five lines of code, executed simultaneously by thousands of nodes around the world, and you have the beginnings of a solution to a major problem in cryptography: human-friendly authentication. it is important to point out that when the original version of ethereum’s scripting code was designed we did not have name registration in mind; rather, the fact that this is possible came about as an emergent property of its turing-completeness. hopefully this will give you an insight into exactly what ethereum makes possible; for more applications, with code, see the whitepaper. just a few examples include: user-issued currencies / “colored coins” decentralized exchange financial contracts, including leverage trading and hedging crop insurance savings wallets with withdrawal limits peer to peer gambling decentralized dropbox-style data storage decentralized autonomous organizations perhaps now you see why we are excited. who is ethereum the core ethereum team includes four members:   vitalik buterin vitalik buterin first joined the bitcoin community in march 2011, and co-founded bitcoin magazine with mihai alisie in september 2011. he was admitted to the university of waterloo to study computer science in 2012, and in 2013 made the decision to leave waterloo to travel through bitcoin communities around the world and work on bitcoin projects full time. vitalik is responsible for a number of bitcoin projects, including pybitcointools, a fork of bitcoinjsand multisig.info; now, he has returned to canada and is fully dedicated to working on ethereum. mihai alisie mihai alisie’s first foray into the bitcoin community is bitcoin magazine, in september 2011. from issue #1, which was shipped from his living room in romania, to today bitcoin magazine bears mihai’s imprint, and has grown as he has grown with the magazine. what started out as a team of people that didn’t have any experience in the publishing industry, is now distributing a physical magazine internationally and in barnes & noble bookstores across the us. mihai is also involved in an innovative online e-commerce startup known as egora. anthony di iorio anthony di iorio is a founding member, board member & executive director of the bitcoin alliance of canada, founder of the toronto bitcoin meetup group, and partner / founder in various bitcoin start-ups and initiatives including the in-browser bitcoin wallet kryptokit, cointalk, the toronto-based bitcoin hub and coworking spacebitcoin decentral, bitcoin across america, and the global bitcoin alliance. charles hoskinson charles hoskinson is an entrepreneur and cryptographer actively working on ventures in the bitcoin ecosystem. he founded both the bitcoin education project and invictus innovations prior to accepting his current role as a core developer of the ethereum project. he studied at metropolitan state university of denver and university of colorado at boulder with an emphasis in analytic number theory. charles is known for his love of economics, horology and moocs alongside a passion for chess and games of strategy. we also have an excellent team of developers, entrepreneurs, marketers and evangelists: dr. gavin wood: core c++ developer geff obscura: core go developer dr. emanuele costa: quantitative analyst; scrum master joseph lubin: software engineering, quantitative analyst eric lombrozo: software architect max kaye: developer jonathan mohan: media, marketing and evangelism (bitcoinnyc) wendell davis: strategic partner and branding (hive wallet) anthony donofrio: logos, branding, web development (hive wallet) taylor gerring: web development paul snow: language development, software development chris odom: strategic partner, developer (open transactions) jerry liu and bin lu: chinese strategy and translations (http://www.8btc.com/ethereum) hai nguyen: accounting amir shetrit: business development (colored coins) steve dakh: developer (kryptokit) kyle kurbegovich: media (cointalk) looking forward i personally will be presenting at the bitcoin conference in miami on jan 25-26. soon after that, on february 1, the ether pre-sale will begin, at which point anyone will be able to obtain some of the initial pre-allocated ether (ethereum’s internal currency) at a rate of 1000-2000 ether for 1 btc by going to http://fund.ethereum.org. the pre-sale will run throughout february and march, and early funders will get higher rewards; anyone who sends money in the first seven days will receive the full 2000 ether, then 1980 ether on the 8th day, 1960 on the 9th day, and so forth until the baseline rate of 1000 ether per btc is retained for the last three days of the pre-sale. we will be able to develop fully functional and robust ethereum clients with as little as 500 btc funding with current rates; basic implementations in go, c++ and python are coming close to testnet quality already. however, we are seeking to go much further than that. ethereum is not “just another altcoin”; it is a new way forward for cryptocurrency, and ultimately for peer-to-peer protocols as a whole. to that end, we would like to be able to invest a large quantity of funds into securing top-notch talent for improving the security and scalability of the ethereum network itself, but also supporting a robust ethereum ecosystem hopefully bringing other cryptocurrency and peer-to-peer projects into our fold. we are already well underway in talks with kryptokit, humint and opentransactions, and are interested in working with other groups such as tahoe-lafs, bitmessage and bitcloud as well. all of these projects can potentially benefit from integrating with the ethereum blockchain in some fashion, simply because the layer is so universal; because of its turing-completeness, an ethereum contract can be constructed to incentivize nearly everything, and even entirely non-financial uses such as public key registration have extremely wide-reaching benefits for any decentralized cryptographic product that intends to include, for example, a social network. all of these projects will add great value to the ethereum ecosystem, and the ethereum ecosystem will hopefully add great value to them. we do not wish to compete with any organization; we intend to work together. throughout the fundraiser, we will be working hard on development; we will release a centralized testnet, a server to which anyone can push contracts and transactions, very soon, and will then follow up with a decentralized testnet to test networking features and mining algorithms. we also intend to host a contest, similar to those used to decide the algorithms for the advanced encryption standard (aes) in 2005 and sha3 in 2013, in which we invite researchers from universities around the world to compete to develop the best possible specialized hardware-resistant, centralization-resistant and fair mining algorithms, and will also explore alternatives such as proof of stake, proof of burn and proof of excellence. details on this will be further released in february. finally, to promote local community development, we also intend to create public community hubs and incubators, which we are tentatively calling “holons”, in several cities around the world. the first holon will be based inside of bitcoin decentral in toronto, and a substantial portion of ethereum development will take place there; anyone who is seriously interested in participating heavily in ethereum should consider giving us a visit over the next month. other cities we are looking into include san francisco, amsterdam, tel aviv and some city in asia; this part of the project is still in a very early phase of development, and more details will come over the next month. for now feel free to check out our resources: forum: http://forum.ethereum.org blog: http://blog.ethereum.org wiki: http://wiki.ethereum.org whitepaper: https://ethereum.org/ethereum.html reddit: http://reddit.com/r/ethereum previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle fast fourier transforms 2019 may 12 see all posts trigger warning: specialized mathematical topic special thanks to karl floersch for feedback one of the more interesting algorithms in number theory is the fast fourier transform (fft). ffts are a key building block in many algorithms, including extremely fast multiplication of large numbers, multiplication of polynomials, and extremely fast generation and recovery of erasure codes. erasure codes in particular are highly versatile; in addition to their basic use cases in fault-tolerant data storage and recovery, erasure codes also have more advanced use cases such as securing data availability in scalable blockchains and starks. this article will go into what fast fourier transforms are, and how some of the simpler algorithms for computing them work. background the original fourier transform is a mathematical operation that is often described as converting data between the "frequency domain" and the "time domain". what this means more precisely is that if you have a piece of data, then running the algorithm would come up with a collection of sine waves with different frequencies and amplitudes that, if you added them together, would approximate the original data. fourier transforms can be used for such wonderful things as expressing square orbits through epicycles and deriving a set of equations that can draw an elephant: ok fine, fourier transforms also have really important applications in signal processing, quantum mechanics, and other areas, and help make significant parts of the global economy happen. but come on, elephants are cooler. running the fourier transform algorithm in the "inverse" direction would simply take the sine waves and add them together and compute the resulting values at as many points as you wanted to sample. the kind of fourier transform we'll be talking about in this post is a similar algorithm, except instead of being a continuous fourier transform over real or complex numbers, it's a discrete fourier transform over finite fields (see the "a modular math interlude" section here for a refresher on what finite fields are). instead of talking about converting between "frequency domain" and "time domain", here we'll talk about two different operations: multi-point polynomial evaluation (evaluating a degree \(< n\) polynomial at \(n\) different points) and its inverse, polynomial interpolation (given the evaluations of a degree \(< n\) polynomial at \(n\) different points, recovering the polynomial). for example, if we are operating in the prime field with modulus 5, then the polynomial \(y = x² + 3\) (for convenience we can write the coefficients in increasing order: \([3,0,1]\)) evaluated at the points \([0,1,2]\) gives the values \([3,4,2]\) (not \([3, 4, 7]\) because we're operating in a finite field where the numbers wrap around at 5), and we can actually take the evaluations \([3,4,2]\) and the coordinates they were evaluated at (\([0,1,2]\)) to recover the original polynomial \([3,0,1]\). there are algorithms for both multi-point evaluation and interpolation that can do either operation in \(o(n^2)\) time. multi-point evaluation is simple: just separately evaluate the polynomial at each point. here's python code for doing that: def eval_poly_at(self, poly, x, modulus): y = 0 power_of_x = 1 for coefficient in poly: y += power_of_x * coefficient power_of_x *= x return y % modulus the algorithm runs a loop going through every coefficient and does one thing for each coefficient, so it runs in \(o(n)\) time. multi-point evaluation involves doing this evaluation at \(n\) different points, so the total run time is \(o(n^2)\). lagrange interpolation is more complicated (search for "lagrange interpolation" here for a more detailed explanation). the key building block of the basic strategy is that for any domain \(d\) and point \(x\), we can construct a polynomial that returns \(1\) for \(x\) and \(0\) for any value in \(d\) other than \(x\). for example, if \(d = [1,2,3,4]\) and \(x = 1\), the polynomial is: \[ y = \frac{(x-2)(x-3)(x-4)}{(1-2)(1-3)(1-4)} \] you can mentally plug in \(1\), \(2\), \(3\) and \(4\) to the above expression and verify that it returns \(1\) for \(x= 1\) and \(0\) in the other three cases. we can recover the polynomial that gives any desired set of outputs on the given domain by multiplying and adding these polynomials. if we call the above polynomial \(p_1\), and the equivalent ones for \(x=2\), \(x=3\), \(x=4\), \(p_2\), \(p_3\) and \(p_4\), then the polynomial that returns \([3,1,4,1]\) on the domain \([1,2,3,4]\) is simply \(3 \cdot p_1 + p_2 + 4 \cdot p_3 + p_4\). computing the \(p_i\) polynomials takes \(o(n^2)\) time (you first construct the polynomial that returns to 0 on the entire domain, which takes \(o(n^2)\) time, then separately divide it by \((x x_i)\) for each \(x_i\)), and computing the linear combination takes another \(o(n^2)\) time, so it's \(o(n^2)\) runtime total. what fast fourier transforms let us do, is make both multi-point evaluation and interpolation much faster. fast fourier transforms there is a price you have to pay for using this much faster algorithm, which is that you cannot choose any arbitrary field and any arbitrary domain. whereas with lagrange interpolation, you could choose whatever x coordinates and y coordinates you wanted, and whatever field you wanted (you could even do it over plain old real numbers), and you could get a polynomial that passes through them., with an fft, you have to use a finite field, and the domain must be a multiplicative subgroup of the field (that is, a list of powers of some "generator" value). for example, you could use the finite field of integers modulo \(337\), and for the domain use \([1, 85, 148, 111, 336, 252, 189, 226]\) (that's the powers of \(85\) in the field, eg. \(85^3\) % \(337 = 111\); it stops at \(226\) because the next power of \(85\) cycles back to \(1\)). futhermore, the multiplicative subgroup must have size \(2^n\) (there's ways to make it work for numbers of the form \(2^{m} \cdot 3^n\) and possibly slightly higher prime powers but then it gets much more complicated and inefficient). the finite field of intergers modulo \(59\), for example, would not work, because there are only multiplicative subgroups of order \(2\), \(29\) and \(58\); \(2\) is too small to be interesting, and the factor \(29\) is far too large to be fft-friendly. the symmetry that comes from multiplicative groups of size \(2^n\) lets us create a recursive algorithm that quite cleverly calculate the results we need from a much smaller amount of work. to understand the algorithm and why it has a low runtime, it's important to understand the general concept of recursion. a recursive algorithm is an algorithm that has two cases: a "base case" where the input to the algorithm is small enough that you can give the output directly, and the "recursive case" where the required computation consists of some "glue computation" plus one or more uses of the same algorithm to smaller inputs. for example, you might have seen recursive algorithms being used for sorting lists. if you have a list (eg. \([1,8,7,4,5,6,3,2,9]\)), then you can sort it using the following procedure: if the input has one element, then it's already "sorted", so you can just return the input. if the input has more than one element, then separately sort the first half of the list and the second half of the list, and then merge the two sorted sub-lists (call them \(a\) and \(b\)) as follows. maintain two counters, \(apos\) and \(bpos\), both starting at zero, and maintain an output list, which starts empty. until either \(apos\) or \(bpos\) is at the end of the corresponding list, check if \(a[apos]\) or \(b[bpos]\) is smaller. whichever is smaller, add that value to the end of the output list, and increase that counter by \(1\). once this is done, add the rest of whatever list has not been fully processed to the end of the output list, and return the output list. note that the "glue" in the second procedure has runtime \(o(n)\): if each of the two sub-lists has \(n\) elements, then you need to run through every item in each list once, so it's \(o(n)\) computation total. so the algorithm as a whole works by taking a problem of size \(n\), and breaking it up into two problems of size \(\frac{n}{2}\), plus \(o(n)\) of "glue" execution. there is a theorem called the master theorem that lets us compute the total runtime of algorithms like this. it has many sub-cases, but in the case where you break up an execution of size \(n\) into \(k\) sub-cases of size \(\frac{n}{k}\) with \(o(n)\) glue (as is the case here), the result is that the execution takes time \(o(n \cdot log(n))\). an fft works in the same way. we take a problem of size \(n\), break it up into two problems of size \(\frac{n}{2}\), and do \(o(n)\) glue work to combine the smaller solutions into a bigger solution, so we get \(o(n \cdot log(n))\) runtime total much faster than \(o(n^2)\). here is how we do it. i'll describe first how to use an fft for multi-point evaluation (ie. for some domain \(d\) and polynomial \(p\), calculate \(p(x)\) for every \(x\) in \(d\)), and it turns out that you can use the same algorithm for interpolation with a minor tweak. suppose that we have an fft where the given domain is the powers of \(x\) in some field, where \(x^{2^{k}} = 1\) (eg. in the case we introduced above, the domain is the powers of \(85\) modulo \(337\), and \(85^{2^{3}} = 1\)). we have some polynomial, eg. \(y = 6x^7 + 2x^6 + 9x^5 + 5x^4 + x^3 + 4x^2 + x + 3\) (we'll write it as \(p = [3, 1, 4, 1, 5, 9, 2, 6]\)). we want to evaluate this polynomial at each point in the domain, ie. at each of the eight powers of \(85\). here is what we do. first, we break up the polynomial into two parts, which we'll call \(evens\) and \(odds\): \(evens = [3, 4, 5, 2]\) and \(odds = [1, 1, 9, 6]\) (or \(evens = 2x^3 + 5x^2 + 4x + 3\) and \(odds = 6x^3 + 9x^2 + x + 1\); yes, this is just taking the even-degree coefficients and the odd-degree coefficients). now, we note a mathematical observation: \(p(x) = evens(x^2) + x \cdot odds(x^2)\) and \(p(-x) = evens(x^2) x \cdot odds(x^2)\) (think about this for yourself and make sure you understand it before going further). here, we have a nice property: \(evens\) and \(odds\) are both polynomials half the size of \(p\), and furthermore, the set of possible values of \(x^2\) is only half the size of the original domain, because there is a two-to-one correspondence: \(x\) and \(-x\) are both part of \(d\) (eg. in our current domain \([1, 85, 148, 111, 336, 252, 189, 226]\), 1 and 336 are negatives of each other, as \(336 = -1\) % \(337\), as are \((85, 252)\), \((148, 189)\) and \((111, 226)\). and \(x\) and \(-x\) always both have the same square. hence, we can use an fft to compute the result of \(evens(x)\) for every \(x\) in the smaller domain consisting of squares of numbers in the original domain (\([1, 148, 336, 189]\)), and we can do the same for odds. and voila, we've reduced a size-\(n\) problem into half-size problems. the "glue" is relatively easy (and \(o(n)\) in runtime): we receive the evaluations of \(evens\) and \(odds\) as size-\(\frac{n}{2}\) lists, so we simply do \(p[i] = evens\_result[i] + domain[i]\cdot odds\_result[i]\) and \(p[\frac{n}{2} + i] = evens\_result[i] domain[i]\cdot odds\_result[i]\) for each index \(i\). here's the full code: def fft(vals, modulus, domain): if len(vals) == 1: return vals l = fft(vals[::2], modulus, domain[::2]) r = fft(vals[1::2], modulus, domain[::2]) o = [0 for i in vals] for i, (x, y) in enumerate(zip(l, r)): y_times_root = y*domain[i] o[i] = (x+y_times_root) % modulus o[i+len(l)] = (x-y_times_root) % modulus return o we can try running it: >>> fft([3,1,4,1,5,9,2,6], 337, [1, 85, 148, 111, 336, 252, 189, 226]) [31, 70, 109, 74, 334, 181, 232, 4] and we can check the result; evaluating the polynomial at the position \(85\), for example, actually does give the result \(70\). note that this only works if the domain is "correct"; it needs to be of the form \([x^i\) % \(modulus\) for \(i\) in \(range(n)]\) where \(x^n = 1\). an inverse fft is surprisingly simple: def inverse_fft(vals, modulus, domain): vals = fft(vals, modulus, domain) return [x * modular_inverse(len(vals), modulus) % modulus for x in [vals[0]] + vals[1:][::-1]] basically, run the fft again, but reverse the result (except the first item stays in place) and divide every value by the length of the list. >>> domain = [1, 85, 148, 111, 336, 252, 189, 226] >>> def modular_inverse(x, n): return pow(x, n 2, n) >>> values = fft([3,1,4,1,5,9,2,6], 337, domain) >>> values [31, 70, 109, 74, 334, 181, 232, 4] >>> inverse_fft(values, 337, domain) [3, 1, 4, 1, 5, 9, 2, 6] now, what can we use this for? here's one fun use case: we can use ffts to multiply numbers very quickly. suppose we wanted to multiply \(1253\) by \(1895\). here is what we would do. first, we would convert the problem into one that turns out to be slightly easier: multiply the polynomials \([3, 5, 2, 1]\) by \([5, 9, 8, 1]\) (that's just the digits of the two numbers in increasing order), and then convert the answer back into a number by doing a single pass to carry over tens digits. we can multiply polynomials with ffts quickly, because it turns out that if you convert a polynomial into evaluation form (ie. \(f(x)\) for every \(x\) in some domain \(d\)), then you can multiply two polynomials simply by multiplying their evaluations. so what we'll do is take the polynomials representing our two numbers in coefficient form, use ffts to convert them to evaluation form, multiply them pointwise, and convert back: >>> p1 = [3,5,2,1,0,0,0,0] >>> p2 = [5,9,8,1,0,0,0,0] >>> x1 = fft(p1, 337, domain) >>> x1 [11, 161, 256, 10, 336, 100, 83, 78] >>> x2 = fft(p2, 337, domain) >>> x2 [23, 43, 170, 242, 3, 313, 161, 96] >>> x3 = [(v1 * v2) % 337 for v1, v2 in zip(x1, x2)] >>> x3 [253, 183, 47, 61, 334, 296, 220, 74] >>> inverse_fft(x3, 337, domain) [15, 52, 79, 66, 30, 10, 1, 0] this requires three ffts (each \(o(n \cdot log(n))\) time) and one pointwise multiplication (\(o(n)\) time), so it takes \(o(n \cdot log(n))\) time altogether (technically a little bit more than \(o(n \cdot log(n))\), because for very big numbers you would need replace \(337\) with a bigger modulus and that would make multiplication harder, but close enough). this is much faster than schoolbook multiplication, which takes \(o(n^2)\) time: 3 5 2 1 -----------5 | 15 25 10 5 9 | 27 45 18 9 8 | 24 40 16 8 1 | 3 5 2 1 -------------------- 15 52 79 66 30 10 1 so now we just take the result, and carry the tens digits over (this is a "walk through the list once and do one thing at each point" algorithm so it takes \(o(n)\) time): [15, 52, 79, 66, 30, 10, 1, 0] [ 5, 53, 79, 66, 30, 10, 1, 0] [ 5, 3, 84, 66, 30, 10, 1, 0] [ 5, 3, 4, 74, 30, 10, 1, 0] [ 5, 3, 4, 4, 37, 10, 1, 0] [ 5, 3, 4, 4, 7, 13, 1, 0] [ 5, 3, 4, 4, 7, 3, 2, 0] and if we read the digits from top to bottom, we get \(2374435\). let's check the answer.... >>> 1253 * 1895 2374435 yay! it worked. in practice, on such small inputs, the difference between \(o(n \cdot log(n))\) and \(o(n^2)\) isn't that large, so schoolbook multiplication is faster than this fft-based multiplication process just because the algorithm is simpler, but on large inputs it makes a really big difference. but ffts are useful not just for multiplying numbers; as mentioned above, polynomial multiplication and multi-point evaluation are crucially important operations in implementing erasure coding, which is a very important technique for building many kinds of redundant fault-tolerant systems. if you like fault tolerance and you like efficiency, ffts are your friend. ffts and binary fields prime fields are not the only kind of finite field out there. another kind of finite field (really a special case of the more general concept of an extension field, which are kind of like the finite-field equivalent of complex numbers) are binary fields. in an binary field, each element is expressed as a polynomial where all of the entries are \(0\) or \(1\), eg. \(x^3 + x + 1\). adding polynomials is done modulo \(2\), and subtraction is the same as addition (as \(-1 = 1 \bmod 2\)). we select some irreducible polynomial as a modulus (eg. \(x^4 + x + 1\); \(x^4 + 1\) would not work because \(x^4 + 1\) can be factored into \((x^2 + 1)\cdot(x^2 + 1)\) so it's not "irreducible"); multiplication is done modulo that modulus. for example, in the binary field mod \(x^4 + x + 1\), multiplying \(x^2 + 1\) by \(x^3 + 1\) would give \(x^5 + x^3 + x^2 + 1\) if you just do the multiplication, but \(x^5 + x^3 + x^2 + 1 = (x^4 + x + 1)\cdot x + (x^3 + x + 1)\), so the result is the remainder \(x^3 + x + 1\). we can express this example as a multiplication table. first multiply \([1, 0, 0, 1]\) (ie. \(x^3 + 1\)) by \([1, 0, 1]\) (ie. \(x^2 + 1\)): 1 0 0 1 -------1 | 1 0 0 1 0 | 0 0 0 0 1 | 1 0 0 1 ----------- 1 0 1 1 0 1 the multiplication result contains an \(x^5\) term so we can subtract \((x^4 + x + 1)\cdot x\): 1 0 1 1 0 1 1 1 0 0 1 [(x⁴ + x + 1) shifted right by one to reflect being multipled by x] ----------- 1 1 0 1 0 0 and we get the result, \([1, 1, 0, 1]\) (or \(x^3 + x + 1\)). addition and multiplication tables for the binary field mod \(x^4 + x + 1\). field elements are expressed as integers converted from binary (eg. \(x^3 + x^2 \rightarrow 1100 \rightarrow 12\)) binary fields are interesting for two reasons. first of all, if you want to erasure-code binary data, then binary fields are really convenient because \(n\) bytes of data can be directly encoded as a binary field element, and any binary field elements that you generate by performing computations on it will also be \(n\) bytes long. you cannot do this with prime fields because prime fields' size is not exactly a power of two; for example, you could encode every \(2\) bytes as a number from \(0...65536\) in the prime field modulo \(65537\) (which is prime), but if you do an fft on these values, then the output could contain \(65536\), which cannot be expressed in two bytes. second, the fact that addition and subtraction become the same operation, and \(1 + 1 = 0\), create some "structure" which leads to some very interesting consequences. one particularly interesting, and useful, oddity of binary fields is the "freshman's dream" theorem: \((x+y)^2 = x^2 + y^2\) (and the same for exponents \(4, 8, 16...\) basically any power of two). but if you want to use binary fields for erasure coding, and do so efficiently, then you need to be able to do fast fourier transforms over binary fields. but then there is a problem: in a binary field, there are no (nontrivial) multiplicative groups of order \(2^n\). this is because the multiplicative groups are all order \(2^n\)-1. for example, in the binary field with modulus \(x^4 + x + 1\), if you start calculating successive powers of \(x+1\), you cycle back to \(1\) after \(\it 15\) steps not \(16\). the reason is that the total number of elements in the field is \(16\), but one of them is zero, and you're never going to reach zero by multiplying any nonzero value by itself in a field, so the powers of \(x+1\) cycle through every element but zero, so the cycle length is \(15\), not \(16\). so what do we do? the reason we needed the domain to have the "structure" of a multiplicative group with \(2^n\) elements before is that we needed to reduce the size of the domain by a factor of two by squaring each number in it: the domain \([1, 85, 148, 111, 336, 252, 189, 226]\) gets reduced to \([1, 148, 336, 189]\) because \(1\) is the square of both \(1\) and \(336\), \(148\) is the square of both \(85\) and \(252\), and so forth. but what if in a binary field there's a different way to halve the size of a domain? it turns out that there is: given a domain containing \(2^k\) values, including zero (technically the domain must be a subspace), we can construct a half-sized new domain \(d'\) by taking \(x \cdot (x+k)\) for \(x\) in \(d\) using some specific \(k\) in \(d\). because the original domain is a subspace, since \(k\) is in the domain, any \(x\) in the domain has a corresponding \(x+k\) also in the domain, and the function \(f(x) = x \cdot (x+k)\) returns the same value for \(x\) and \(x+k\) so we get the same kind of two-to-one correspondence that squaring gives us. \(x\) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 \(x \cdot (x+1)\) 0 0 6 6 7 7 1 1 4 4 2 2 3 3 5 5 so now, how do we do an fft on top of this? we'll use the same trick, converting a problem with an \(n\)-sized polynomial and \(n\)-sized domain into two problems each with an \(\frac{n}{2}\)-sized polynomial and \(\frac{n}{2}\)-sized domain, but this time using different equations. we'll convert a polynomial \(p\) into two polynomials \(evens\) and \(odds\) such that \(p(x) = evens(x \cdot (k-x)) + x \cdot odds(x \cdot (k-x))\). note that for the \(evens\) and \(odds\) that we find, it will also be true that \(p(x+k) = evens(x \cdot (k-x)) + (x+k) \cdot odds(x \cdot (k-x))\). so we can then recursively do an fft to \(evens\) and \(odds\) on the reduced domain \([x \cdot (k-x)\) for \(x\) in \(d]\), and then we use these two formulas to get the answers for two "halves" of the domain, one offset by \(k\) from the other. converting \(p\) into \(evens\) and \(odds\) as described above turns out to itself be nontrivial. the "naive" algorithm for doing this is itself \(o(n^2)\), but it turns out that in a binary field, we can use the fact that \((x^2-kx)^2 = x^4 k^2 \cdot x^2\), and more generally \((x^2-kx)^{2^{i}} = x^{2^{i+1}} k^{2^{i}} \cdot x^{2^{i}}\) , to create yet another recursive algorithm to do this in \(o(n \cdot log(n))\) time. and if you want to do an inverse fft, to do interpolation, then you need to run the steps in the algorithm in reverse order. you can find the complete code for doing this here: https://github.com/ethereum/research/tree/master/binary_fft, and a paper with details on more optimal algorithms here: http://www.math.clemson.edu/~sgao/papers/gm10.pdf so what do we get from all of this complexity? well, we can try running the implementation, which features both a "naive" \(o(n^2)\) multi-point evaluation and the optimized fft-based one, and time both. here are my results: >>> import binary_fft as b >>> import time, random >>> f = b.binaryfield(1033) >>> poly = [random.randrange(1024) for i in range(1024)] >>> a = time.time(); x1 = b._simple_ft(f, poly); time.time() a 0.5752472877502441 >>> a = time.time(); x2 = b.fft(f, poly, list(range(1024))); time.time() a 0.03820443153381348 and as the size of the polynomial gets larger, the naive implementation (_simple_ft) gets slower much more quickly than the fft: >>> f = b.binaryfield(2053) >>> poly = [random.randrange(2048) for i in range(2048)] >>> a = time.time(); x1 = b._simple_ft(f, poly); time.time() a 2.2243144512176514 >>> a = time.time(); x2 = b.fft(f, poly, list(range(2048))); time.time() a 0.07896280288696289 and voila, we have an efficient, scalable way to multi-point evaluate and interpolate polynomials. if we want to use ffts to recover erasure-coded data where we are missing some pieces, then algorithms for this also exist, though they are somewhat less efficient than just doing a single fft. enjoy! dark mode toggle on medium-of-exchange token valuations 2017 oct 17 see all posts one kind of token model that has become popular among many recent token sale projects is the "network medium of exchange token". the general pitch for this kind of token goes as follows. we, the developers, build a network, and this network allows you to do new cool stuff. this network is a sharing-economy-style system: it consists purely of a set of sellers, that provide resources within some protocol, and buyers that purchase the services, where both buyers and sellers come from the community. but the purchase and sale of things within this network must be done with the new token that we're selling, and this is why the token will have value. if it were the developers themselves that were acting as the seller, then this would be a very reasonable and normal arrangement, very similar in nature to a kickstarter-style product sale. the token actually would, in a meaningful economic sense, be backed by the services that are provided by the developers. we can see this in more detail by describing what is going on in a simple economic model. suppose that \(n\) people value a product that a developer wants to release at \(\$x\), and believe the developer will give them the product. the developer does a sale, and raises \(n\) units for \(\$w < x\) each, thus raising a total revenue of \(\$nw\). the developer builds the product, and gives it to each of the buyers. at the end of the day, the buyers are happy, and the developer is happy. nobody feels like they made an avoidable mistake in participating, and everyone's expectations have been met. this kind of economic model is clearly stable. now, let's look at the story with a "medium of exchange" token. \(n\) people value a product that will exist in a decentralized network at \(\$x\); the product will be sold at a price of \(\$w < x\). they each buy \(\$w\) of tokens in the sale. the developer builds the network. some sellers come in, and offer the product inside the network for \(\$w\). the buyers use their tokens to purchase this product, spending \(\$w\) of tokens and getting \(\$x\) of value. the sellers spend \(\$v < w\) of resources and effort producing this product, and they now have \(\$w\) worth of tokens. notice that here, the cycle is not complete, and in fact it never will be; there needs to be an ongoing stream of buyers and sellers for the token to continue having its value. the stream does not strictly speaking have to be endless; if in every round there is a chance of at least \(\frac{v}{w}\) that there will be a next round, then the model still works, as even though someone will eventually be cheated, the risk of any individual participant becoming that person is lower than the benefit that they get from participating. it's also totally possible that the token would depreciate in each round, with its value multiplying by some factor \(f\) where \(\frac{v}{w} < f < 1\), until it eventually reaches a price of zero, and it would still be on net in everyone's interest to participate. hence, the model is theoretically feasible, but you can see how this model is more complex and more tenuous than the simple "developers as seller" model. traditional macroeconomics has a simple equation to try to value a medium of exchange: \(mv = pt\) here: \(m\) is the total money supply; that is, the total number of coins \(v\) is the "velocity of money"; that is, the number of times that an average coin changes hands every day \(p\) is the "price level". this is the price of goods and services in terms of the token; so it is actually the inverse of the currency's price \(t\) is the transaction volume: the economic value of transactions per day the proof for this is a trivial equality: if there are \(n\) coins, and each changes hands \(m\) times per day, then this is \(m \cdot n\) coins' worth of economic value transacted per day. if this represents \(\$t\) worth of economic value, then the price of each coin is \(\frac{t}{m \cdot n}\), so the "price level" is the inverse of this, \(\frac{m \cdot n}{t}\). for easier analysis, we can recast two variables: we refer to \(\frac{1}{v}\) with \(h\), the time that a user holds a coin before using it to make a transaction we refer to \(\frac{1}{p}\) with \(c\), the price of the currency (think \(c = cost\)) now, we have: \(\frac{m}{h} = \frac{t}{c}\) \(mc = th\) the left term is quite simply the market cap. the right term is the economic value transacted per day, multiplied by the amount of time that a user holds a coin before using it to transact. this is a steady-state model, assuming that the same quantity of users will also be there. in reality, however, the quantity of users may change, and so the price may change. the time that users hold a coin may change, and this may cause the price to change as well. let us now look once again at the economic effect on the users. what do users lose by using an application with a built-in appcoin rather than plain old ether (or bitcoin, or usd)? the simplest way to express this is as follows: the "implicit cost" imposed by such a system on users the cost to the user of holding those coins for that period of time, instead of holding that value in the currency that they would otherwise have preferred to hold. there are many factors involved in this cost: cognitive costs, exchange costs and spreads, transaction fees, and many smaller items. one particular significant factor of this implicit cost is expected return. if a user expects the appcoin to only grow in value by 1% per year, while their other available alternatives grow 3% per year, and they hold $20 of the currency for five days, then that is an expected loss of roughly \(\$20 \cdot 2% \cdot 5 / 365 = \$0.0054\). one immediate conclusion from this particular insight is that appcoins are very much a multi-equilibrium game. if the appcoin grows at 2% per year, then the fee drops to $0.0027, and this essentially makes the "de-facto fee" of the application (or at least a large component of it) 2x cheaper, attracting more users and growing its value more. if the appcoin starts falling at 10% per year, however, then the "de-facto fee" grows to $0.035, driving many users away and accelerating its growth. this leads to increased opportunities for market manipulation, as a manipulator would not just be wasting their money fighting against a single equilibrium, but may in fact successfully nudge a given currency from one equilibrium into another, and profit from successfully "predicting" (ie. causing) this shift. it also means there is a large amount of path dependency, and established brands matter a lot; witness the epic battles over which fork of the bitcoin blockchain can be called bitcoin for one particular high-profile example. another, and perhaps even more important, conclusion is that the market cap of an appcoin depends crucially on the holding time \(h\). if someone creates a very efficient exchange, which allows users to purchase an appcoin in real time and then immediately use it in the application, then allowing sellers to immediately cash out, then the market cap would drop precipitously. if a currency is stable or prospects are looking optimistic, then this may not matter because users actually see no disadvantage from holding the token instead of holding something else (ie. zero "de-facto fee"), but if prospects start to turn sour then such a well-functioning exchange can acelerate its demise. you might think that exchanges are inherently inefficient, requiring users to create an account, login, deposit coins, wait for 36 confirmations, trade and logout, but in fact hyper-efficient exchanges are around the corner. here is a thread discussing designs for fully autonomous synchronous on-chain transactions, which can convert token a into token b, and possibly even then use token b to do something, within a single transaction. many other platforms are being developed as well. what this all serves to show is that relying purely on the medium-of-exchange argument to support a token value, while attractive because of its seeming ability to print money out of thin air, is ultimately quite brittle. protocol tokens using this model may well be sustained for some time due to irrationality and temporary equilibria where the implicit cost of holding the token is zero, but it is a kind of model which always has an unavoidable risk of collapsing at any time. so what is the alternative? one simple alternative is the etherdelta approach, where an application simply collects fees in the interface. one common criticism is: but can't someone fork the interface to take out the fees? a counter-retort is: someone can also fork the interface to replace your protocol token with eth, btc, doge or whatever else users would prefer to use. one can make a more sophisticated argument that this is hard because the "pirate" version would have to compete with the "official" version for network effect, but one can just as easily create an official fee-paying client that refuses to interact with non-fee-paying clients as well; this kind of network effect-based enforcement is similar to how value-added-taxes are typically enforced in europe and other places. official-client buyers would not interact with non-official-client sellers, and official-client sellers would not interact with non-official-client buyers, so a large group of users would need to switch to the "pirate" client at the same time to successfully dodge fees. this is not perfectly robust, but it is certainly as good as the approach of creating a new protocol token. if developers want to front-load revenue to fund initial development, then they can sell a token, with the property that all fees paid are used to buy back some of the token and burn it; this would make the token backed by the future expected value of upcoming fees spent inside the system. one can transform this design into a more direct utility token by requiring users to use the utility token to pay fees, and having the interface use an exchange to automatically purchase tokens if the user does not have tokens already. the important thing is that for the token to have a stable value, it is highly beneficial for the token supply to have sinks places where tokens actually disappear and so the total token quantity decreases over time. this way, there is a more transparent and explicit fee paid by users, instead of the highly variable and difficult to calculate "de-facto fee", and there is also a more transparent and explicit way to figure out what the value of protocol tokens should be. the question of mining | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the question of mining posted by vitalik buterin on march 20, 2014 research & development there are a lot of interesting changes to the ethereum protocol that are in the works, which will hopefully improve the power of the system, add further features such as light-client friendliness and a higher degree of extensibility, and make ethereum contracts easier to code. theoretically, none of these changes are necessary; the ethereum protocol is fine as it stands today, and can theoretically be released as is once the clients are further built up somewhat; rather, the changes are there to make ethereum better. however, there is one design objective of ethereum where the light at the end of the tunnel is a bit further: mining decentralization. although we always have the backup option of simply sticking with dagger, slasher or sha3, it is entirely unclear that any of those algorithms can truly remain decentralized and mining pool and asic-resistant in the long term (slasher is guaranteed to be decentralized because it’s proof of stake, but has its own moderately problematic flaws). the basic idea behind the mining algorithm that we want to use is essentially in place; however, as in many cases, the devil is in the details. this version of the ethereum mining algorithm is a hashcash-based implementation, similar to bitcoin’s sha256 and litecoin’s scrypt; the idea is for the miner to repeatedly compute a pseudorandom function on a block and a nonce, trying a different nonce each time, until eventually some nonce produces a result which starts with a large number of zeroes. the only room to innovate in this kind of implementation is changing the function; in ethereum’s case, the rough outline of the function, taking the blockchain state (defined as the header, the current state tree, and all the data of the last 16 blocks), is as follows: let h[i] = sha3(sha3(block_header) ++ nonce ++ i) for 0 &lt;= i &lt;= 15 let s be the blockchain state 16 blocks ago. let c[i] be the transaction count of the block i blocks ago. let t[i] be the (h[i] mod c[i])th transaction from the block i blocks ago. apply t[0], t[1] … t[15] sequentially to s. however, every time the transaction leads to processing a contract, (pseudo-)randomly make minor modifications to the code of all contracts affected. let s' be the resulting state. let r be the sha3 of the root of s'. if r &lt;= 2^256 / diff, then nonce is a valid nonce. to summarize in non-programmatic language, the mining algorithm requires the miner to grab a few random transactions from the last 16 blocks, run the computation of applying them to the state 16 blocks ago with a few random modifications, and then take the hash of the result. every new nonce that the miner tries, the miner would have to repeat this process over again, with a new set of random transactions and modifications each time. the benefits of this are: it requires the entire blockchain state to mine, essentially requiring every miner to be a full node. this helps with network decentralization, because a larger number of full nodes exist. because every miner is now required to be a full node, mining pools become much less useful. in the bitcoin world, mining pools serve two key purposes. first, pools even out the mining reward; instead of every block providing a miner with a 0.0001% chance of mining a 16,000block,aminercanmineintothepoolandthepoolgivestheminera116,000 block, a miner can mine into the pool and the pool gives the miner a 1% chance of receiving a payout of 16,000block,aminercanmineintothepoolandthepoolgivestheminera11.60. second, however, pools also provide centralized block validation. instead of having to run a full bitcoin client themselves, a miner can simply grab block header data from the pool and mine using that data without actually verifying the block for themselves. with this algorithm, the second argument is moot, and the first concern can be adequately met by peer-to-peer pools that do not give control of a significant portion of network hashpower to a centralized service. it's asic-resistant almost by definition. because the evm language is turing-complete, any kind of computation that can be done in a normal programming language can be encoded into evm code. therefore, an asic that can run all of evm is by necessity an asic for generalized computation – in other words, a cpu. this also has a primecoin-like social benefit: effort spent toward building evm asics also havs the side benefit of building hardware to make the network faster. the algorithm is relatively computationally quick to verify, although there is no “nice” verification formula that can be run inside evm code. however, there are still several major challenges that remain. first, it is not entirely clear that the system of picking random transactions actually ends up requiring the miner to use the entire blockchain. ideally, the blockchain accesses would be random; in such a setup, a miner with half the blockchain would succeed only on about 1 in 216 nonces. in reality, however, 95% of all transactions will likely use 5% of the blockchain; in such a system, a node with 5% of the memory will only take a slowdown penalty of about 2x. second, and more importantly, however, it is difficult to say how much an evm miner could be optimized. the algorithm definition above asks the miner to “randomly make minor modifications” to the contract. this part is crucial. the reason is this: most transactions have results that are independent of each other; the transactions might be of the form “a sends to b”, “c sends to d”, “e sends to contract f that affects g and h”, etc, with no overlap. hence, without random modification there would be little need for an evm miner to actually do much computation; the computation would happen once, and then the miner would just precompute and store the deltas and apply them immediately. the random modifications mean that the miner has to actually make new evm computations each time the algorithm is run. however, this solution is itself imperfect in two ways. first of all, random modifications can potentially easily result in what would otherwise be very complex and intricate calculations simply ending early, or at least calulations for which the optimizations are very different from the optimizations applied to standard transactions. second, mining algorithms may deliberately skip complex contracts in favor of simple or easily optimizable ones. there are heuristic tricks for battling both problems, but it is entirely unclear exactly what those heuristics would be. another interesting point in favor of this kind of mining is that even if optimized hardware miners emerge, the community has the ability to work together to essentially change the mining algorithm by “poisoning” the transaction pool. engineers can analyze existing asics, determine what their optimizations are, and dump transactions into the blockchain that such optimizations simply do not work with. if 5% of all transactions are effectively poisoned, then asics cannot possibly have a speedup of more than 20x. the nice thing is that there is a reason why people would pay the transaction fees to do this: each individual asic company has the incentive to poison the well for its competitors. these are all challenges that we will be working on heavily in the next few months. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements schellingcoin: a minimal-trust universal data feed | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search schellingcoin: a minimal-trust universal data feed posted by vitalik buterin on march 28, 2014 research & development one of the main applications of ethereum that people have been interested in is financial contracts and derivatives. although financial derivatives have acquired a reputation as a highly risky and destabilizing device with the sole function of enriching speculators, the underlying concept in fact has a number of legitimate uses, some of which actually help people protect themselves against the volatility of financial markets. the main idea here is called “hedging”, and is best explained in the context of bitcoin, where ordinary businesses and individuals with no desire to take massive risks end up needing to deal with high volumes of a risky asset (btc). hedging works as follows. suppose that jane is a business owner who accepts bitcoin for payments and uses it to pay employees, and on average she expects that she will need to keep 100 btc on hand at any time. sometimes, this amount might change; it could be 20 btc or it could be 160 btc. however, she is not at all excited about the prospect of seeing her btc drop 23% in value in a single day and losing several months worth of salary. currently, the “standard” solution is for jane to set up her business to accept payments via bitpay or coinbase, paying a 1% fee to have the bitcoins instantly converted into money in her bank account. when she wants to pay btc, she would need to buy the bitcoins back and send them out, paying 1% again (if not more). hedging provides a different approach. instead of constantly trading btc back and forth, jane creates an account on a financial derivatives market, and enters into a contract for difference. in this cfd, jane agrees to put in 20000worthofbtc,andgetback(inbtc)20000 worth of btc, and get back (in btc) 20000worthofbtc,andgetback(inbtc)20000 plus 100foreverydollarthatthebtcpricedrops.ifthebtcpricerises,sheloses100 for every dollar that the btc price drops. if the btc price rises, she loses 100foreverydollarthatthebtcpricedrops.ifthebtcpricerises,sheloses100 per dollar. thus, if the value of one bitcoin decreases by 45,janewouldlose45, jane would lose 45,janewouldlose4500 in the value of her bitcoins, but she would gain $4500 in the cfd. of course, the money does not come out of nowhere; on the other side of the contract is a speculator, betting that the price of btc will go up, and if it does then jane will gain in the value of btc and lose in the cfd, and the speculator would gain in the cfd. given this basic ingredient, jane has three strategies for using it to manage risk: she can keep the cfd at $100 to $1 forever, and if her exposure is off by some amount then she can take that smaller risk. jane can have a bot constantly adjust the cfd to her supply of btc on hand, paying some fees for this but not nearly as much as with bitpay and coinbase. thanks to the magic of ethereum contracts, she can make a cfd that automatically listens to her account balance and retargets itself to her balance, forcing the speculator to assume whatever exposure she needs (within limits), and the speculator will participate in many such contracts to even out their exposure so how do we do cfds? in ethereum, it’s easy; just write a contract to do what you want. here, i provide a specialized version of a cfd that i am calling a “hedging contract”, which acts as a pure self-contained store of value: you put 1000 ether in, you get the same usd value of ether out (unless the value of ether drops so much that the entire contract doesn’t have enough to cover you, in which case you gain the right to immediately withdraw everything and enter into a new hedging contract): if contract.storage[1000] == 0: if tx.value < 1000 _ 10^18: stop contract.storage[1000] = 1 contract.storage[1001] = 998 _ block.contractstorage(d)[i] contract.storage[1002] = block.timestamp + 30 * 86400 contract.storage[1003] = tx.sender else: ethervalue = contract.storage[1001] / block.contractstorage(d)[i] if ethervalue >= 5000: mktx(contract.storage[1003],5000 * 10^18,0,0) else if block.timestamp > contract.storage[1002]: mktx(contract.storage[1003],ethervalue _ 10^18,0,0) mktx(a,(5000 ethervalue) _ 10^18,0,0) if you understand eth-hll, you can figure that example out, and if you can’t it basically does what the description says (the speculator puts up the contract with 4000 eth, the counterparty enters into it with 1000 eth, and there’s an expiry date after 30 days after which anyone can “ping” the contract to return $x worth of eth to the counterparty and the rest to the speculator). we’ll release better eth-hll guides soon, but for now understanding the fine details of the contract is not necessary. however, all of this has a problem: it requires some trusted source from which to grab the price of eth/usd. this is much less of a problem than the other approach, involving trusted to create usd-backed cryptographic assets, because it requires much less infrastructure and the incentive to cheat is much smaller, but from a cryptographic purist standpoint it’s not perfect. the fundamental problem is this: cryptography alone has no way of finding out that much about the outside world. you can learn a bit about computational power through proof of work, and you can get some market data between one crypto-asset and another by having an on-chain market, but ultimately there is no term in mathematical algorithms for something like the temperature in berlin. there is no inherent way cryptography can tell you whether the correct answer is 11′c, 17′c or 2725′c; you need human judgement for that (or thermometers, but then you need human judgement to determine which thermometers are trustworthy). schelling time here, i provide a mechanism that allows you to create a decentralized data feed. the economics of it are not perfect, and if large collusions are possible then it may break down, but it is likely close to the best that we can do. in this case, we will use the price of eth/usd as an example; the temperature in berlin, the world gdp or even the result of a computation that does not lend itself to efficient verifiability can also be used. the mechanism relies on a concept known as schelling points. the way it works is at follows. suppose you and another prisoner are kept in separate rooms, and the guards give you two identical pieces of paper with a few numbers on them. if both of you choose the same number, then you will be released; otherwise, because human rights are not particularly relevant in the land of game theory, you will be thrown in solitary confinement for the rest of your lives. the numbers are as follows: 14237 59049 76241 81259 90215 100000 132156 157604 which number do you pick? in theory, these are all arbitrary numbers, and you will pick a random one and have a probability of 1/8 of choosing the same one and getting out of prison. in practice, however, the probability is much higher, because most people choose 100000. why 100000? because each prisoner believes that the number 100000 is somehow “special”, and each prisoner believes that the other believes that 100000 is “special”, and so forth infinitely recursively – an instance ofcommon knowledge. thus each prisoner, believing that the other is more likely to choose 100000, will choose 100000 themselves. obviously, this is an infinitely recursive chain of logic that’s not ultimately “backed” by anything except itself, but cryptocurrency users reading this article should by now be very comfortable with relying on such concepts. this mechanism is how schellingcoin works. the basic protocol is as follows: during an even-numbered block, all users can submit a hash of the eth/usd price together with their ethereum address during the block after, users can submit the value whose hash they provided in the previous block. define the “correctly submitted values” as all values n where h(n+addr) was submitted in the first block and n was submitted in the second block, both messages were signed/sent by the account with address addr and addr is one of the allowed participants in the system. sort the correctly submitted values (if many values are the same, have a secondary sort by h(n+prevhash+addr) whereprevhash is the hash of the last block) every user who submitted a correctly submitted value between the 25th and 75th percentile gains a reward of n tokens (which we’ll call “schells”) the protocol does not include a specific mechanism for preventing sybil attacks; it is assumed that proof of work, proof of stake or some other similar solution will be used. so why does this work? essentially, for the same reason why the prisoner example above worked; the truth is arguably the most powerful schelling point out there. everyone wants to provide the correct answer because everyone expects that everyone else will provide the correct answer and the protocol encourages everyone to provide what everyone else provides. criminal investigators have been using schellingcoin for centuries, putting prisoners into separate rooms and asking them all for their stories on what happened at a given event, relying on the fact that it’s easy to be consistent with many other people if you tell the truth but nearly impossible to coordinate on any specific lie. problems and limits what are the vulnerabilities? in general, collusion attacks. most trivially, if any entity controls more than 50% of all votes, they can basically unilaterally set the median to whatever they want. on the other hand, if there are a near-infinite number of discrete non-communicating entities, then each individual entity has essentially zero impact on the result; realistically, there will be many entities giving the exact same value so there will not even be an opportunity to adjust the result slightly by voting falsely. however, in the middle it gets hazy. if one entity controls 49% of votes, they can all pre-announce that they will vote for some false value, and others will also go with that value out of fear that everyone else will and if they don’t they will be left out. but here is the really fun part: even if one entity controls 1% of votes, if that entity pre-announces some false value that they will vote for and announces that they will give 0.00001 schells to whoever votes for that value, then there are now two schelling points: the truth and the entity’s value. however, the entity’s value contains an incentive to vote for it, so theoretically that schelling point is superior and everyone will go for it instead. in practice, however, this is obviously absurd, in the same category as the famous result that in a prisoner’s dilemma with a preset finite number of rounds the optimal strategy is to cheat every round; the argument is that on the last round there’s no room to punish cheating, so the incentive is to cheat, on the second last round both players know that the other will cheat on the next round for that reason anyway so the incentive is to cheat, and so on recursively to the first round. in practice, people are not capable of processing arbitrary-depth recursion, and in this case in practice there is a massive coordination problem in unseating the dominant schelling point, which only gets worse because everyone that benefits from the schellingcoin has an incentive to attempt to censor any communication of an attempt to disrupt it. thus, a 49% coalition will likely be able to break schellingcoin, but a 1% coalition will not. where is the middle ground? perhaps only time will tell. another potential concern is micro-cheating. if the underlying datum is a value that frequently makes slight changes, which the price is, then if most participants in the schellingcoin are simultaneously participants in a system that uses that schellingcoin, they may have the incentive to slightly tweak their answers in one direction, trying to keep within the 25/75 boundary but at the same time push the median up (or down) very slightly to benefit themselves. other users will predict the presence of micro-disruption, and will thus tweak their answers in that direction themselves to try to stay within the median. thus, if people think that micro-cheating is possible, then micro-cheating may be possible, and if they do not think so then it will not be – a common result in schelling point schemes. there are two ways of dealing with the problem. first, we can try to define the value very unambiguously – eg. “the last ask price of eth/usd on exchange xyz at a time hh:mm:00″, so that a very large portion of answers end up exactly the same and there is no possibility to move the median at all by micro-cheating. however, this introduces centralization in the definition, so needs to be handled carefully. an alternative is to be coarse-grained, defining “the price of eth/usd rounded to two significant digits”. second, we can simply work hard to make the underlying system for picking users avoid biases, both by being decentralization-friendly (ie. proof-of-stake over proof-of-work) and by including users who are likely to have incentives in opposite directions. thus, if we combine schellingcoin and contracts for difference, what we have is a cryptographic asset that i have previously identified as a holy grail of cryptocurrency: an asset which maintains a stable value and is simultaneously trust-free. trust-free is of course a relative term; given the current distribution of mining pools bitcoin’s “trust-free” voting is far from completely free of any trust, but the challenge is to make the protocol as decentralized and future-proof as we can. many of these “holy grails” are not reachable perfectly; even the ones that we think we’ve already reached we often really haven’t (eg. decentralized sybil attack resistance), but every step toward the ultimate goal counts. mining for schells the interesting part about schellingcoin is that it can be used for more than just price feeds. schellingcoin can tell you the temperature in berlin, the world’s gdp or, most interestingly of all, the result of a computation. some computations can be efficiently verified; for example, if i wanted a number n such that the last twelve digits of 3n are 737543007707, that’s hard to compute, but if you submit the value then it’s very easy for a contract or mining algorithm to verify it and automatically provide a reward. other computations, however, cannot be efficiently verified, and most useful computation falls into the latter category. schellingcoin provides a way of using the network as an actual distributed cloud computing system by copying the work among n parties instead of every computer in the network and rewarding only those who provide the most common result. for added efficiency, a more intricate multi-step protocol can have one node do the computation and only use schellingcoin to “spot-check” only a random 1% of the work, allowing for perhaps less than 2x cryptographic overhead. a deposit requirement and harsh penalties for providing an answer that turns out not to pass scrutiny can be used to limit fraud, and another option is to let anyone redo the work and “suggest” a verification index to the network to apply schellingcoin on if they discover any faults. the protocol described above is not a new idea; as i mentioned earlier, it is simply a generalization of a centuries-old criminal investigation practice, and in fact bitcoin’s mining algorithm basically is a schellingcoin on the order of transactions. but the idea can potentially be taken much further, provided that the flaws prove to be surmountable. schellingcoin for eth/usd can be used to provide a decentralized dollar; schellingcoin for computation can be used to provide distributed aws (albeit with no privacy, but we can wait for efficient obfuscation for that). thanks to: neal koblitz, for suggesting the idea of using a spot checking repeated computation approach to provide a “useful proof of work” david friedman, for introducing me to the concept of schelling points in his “positive account of property rights” thomas schelling, for coming up with the concept in the first place an individual i talked to two months ago whose identity i unfortunately forgot for providing the idea of incorporating schelling schemes into ethereum previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements the business imperative behind the ethereum vision | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search the business imperative behind the ethereum vision posted by william mougayar on may 24, 2015 research & development special thanks to vitalik buterin, gavin wood and jeffrey wilcke for countless revisions feedback, picks at their brains, and helping me read their tea leaves. introduction in the world of cryptography-based computer science, the ethereum technology vision has captivated the imagination of a large number of software developers and technologists who saw its obvious promise. but those same promises and their business interpretations (and implications) have not widely reached, nor been well understood by non-technical audiences. as ethereum nears coming out of the “labs” and into the market, it is even more important that its message be widely understood by the business community. the purpose of this article is to answer why ethereum matters, why non-technical people should care, and why we need another global blockchain, even if bitcoin already is one. these are a few key questions, and for each one of them, the more you dive into the answers, the more you start to understand them, and the more you will appreciate ethereum’s unique position, while learning that its approach has merits and longevity. summary ethereum is an alternative decentralized ledger protocol, not an alternative cryptocurrency. ethereum’s ideological lineage contains as much bittorrent, java and freenet as it does bitcoin. from a product perspective, it is a general-purpose, global blockchain that can govern both financial and non-financial types of application states. in its essence, ethereum powers decentralized business logic, also known as smart contracts, represented as cryptographic "boxes" that contain value and only unlock it if certain conditions are met. this business logic executes on the blockchain cloud (no server hosting is required), and automatically enforces the terms of a given agreement between a number of parties. they are a building block for “ðapps”, the new form of decentralized applications that ethereum excels at. and from a front-end (client) point of view, ethereum has a powerful special-purpose browser enabling users to install and interact with any ðapp in a user-friendly manner. the outcome of this new patchwork of technologies is an emerging web3 infrastructure which ethereum is particularly good at enabling. it is based on a 3-tier architecture, comprising an advanced browser as the client, the blockchain ledger as a shared resource, and a virtual network of computers that run smart business logic programs in a decentralized way. compared to bitcoin, ethereum has built a new crypto-technology based framework that provides even larger gains in ease of development as well as even stronger light-client properties, while at the same time allowing applications to share a viable economic environment, and solid blockchain security. the genius behind ethereum is this magical network of computers that enables a new type of software applications: the truly decentralized ones, based on embedding the logic of trust inside small programs and distributing them to run on its blockchain. this has several implications; for developers, incredible cost savings and efficiency in writing new applications; and for non-technical users, a chance to reimagine existing businesses, or create new opportunities, based on unbundling central functions and relegating them to decentralized constructs. ethereum helps anyone wishing to develop decentralized applications, encode arbitrarily complex contractual business logic, launch autonomous agents, and manage relationships that will be mediated entirely by the blockchain. ethereum is a form of specialized cloud computing, albeit one that allows substantial gains in efficiency and cost-effectiveness, where very strong guarantees of security and reliability are required. and it comes with a complete suite of tools for building applications on it. the ethereum transaction ledger can be used to securely execute a wide variety of services including: voting systems, domain name registries, financial exchanges, crowdfunding platforms, company governance, self-enforcing contracts and agreements, intellectual property, smart property, and distributed autonomous organisations. ethereum is spurring business and societal innovation at the global level, while opening the door to a new class of applications, never seen before. in the long term, the resulting outcome will impact economic and control structures. thousands of entrepreneurs and developers are already creating and implementing new ideas, projects and startups based on ethereum. and for existing organizations, business and it leaders and entrepreneurs are encouraged to explore how ethereum enables them to reengineer or innovate on top of their existing services, in order to remain competitive in the future. understanding ethereum so, why is it so difficult to understand ethereum? for the same reason that it’s difficult to understand bitcoin, and for the same reason the internet was difficult to grasp by the general public until at least 3 years after the beginning of its commercialization. that reason is: they all have multiple functional identities. granted, that answer is not particularly enlightening, but it is factual: bitcoin, ethereum and the internet mean different things for different people. but their narrative is strong, because their impact is multi-faceted and far reaching. so let’s dissect ethereum in order to make it more comprehensible. ethereum vs. bitcoin as a starting point, and only as a reference, we could begin by comparing ethereum to bitcoin, because bitcoin seems to have been understood, at least by those who wished to understand it. at first sight, one can easily be duped by apparent similarities because both bitcoin and ethereum are open source platforms that have four common staples: an underlying cryptocurrency, an inherent blockchain, a decentralized consensus-based proof mechanism, and miners that support the network; therefore they must be similar. but that is a wrong conclusion. as soon as you dig deeper, reality reveals that bitcoin and ethereum are more dissimilar than similar. each one of these 4 staples has a fundamentally different purpose in terms of what it enables, and for that reason, ethereum is headed into a different direction than bitcoin. for background, bitcoin was originally designed as a decentralized cryptocurrency network for transacting monetary value, and the main purpose of the bitcoin blockchain was that backbone of trust for these financial transactions. it was only more recently that bitcoin's blockchain started to find use cases in non-financial applications relating to decentralized trust and as a store of value. therefore, the programmability aspects of the bitcoin blockchain itself were almost an after-thought, although sidechain proposals are hoping to make this programmability a little easier. in contrast, ethereum was conceived from day 1 as a software development platform for decentralized applications, and its blockchain was specifically designed to support executing apps on it. so, the ethereum design had the luxury of learning from bitcoin’s experience and remedied some of its shortcomings. for example, the ethereum blockchain is faster at performing verifications, down to a typical range between 5 and 30 seconds, as compared to bitcoin’s 10 mins. the ethereum paradigm revolves around being a network for powering decentralized applications in need of a deterministic, auditable and predictable compute platform, and that is different from the essence of the bitcoin paradigm whose trivial compute machinery is currency-centric. therefore, you cannot blindly apply bitcoin analogies pertaining to the role of mining, the cryptocurrency usage, and the programmability capabilities of the respective blockchains in order to fully understand ethereum. first, the ethereum’s cryptocurrency (called “ether”) is not similar to the bitcoin currency because its main purpose is not for payment of goods or services, nor to be a "digital gold" version, two features that bitcoin excels at, and that ethereum doesn't even attempt to provide. ether is more like an incentive plumbing in the form of required “crypto-fuel” to pay for the transaction fees necessary to run the various smart business logic programs that users submit to its blockchain. a close analogy is to compare ether to cloud-based computational costs. when you run an application in the cloud, you are charged based on a combination of time, storage, data transfer and computing speed requirements. the novelty with ether-based costing is that you are paying to run the business logic on the blockchain. it’s a form of cloud-based micro-value pricing that un-bundles a layer from the traditional cloud computing stack. in addition to being that network fuel, ether will also be tradeable as a cryptocurrency on a variety of open exchanges, but it is expected that its value appreciation will be more rationally influenced by the volume and richness of transaction demands than by currency speculators (which is the opposite to what has happened to bitcoin). second, the ethereum blockchain is fully programmable by design, and more economically efficient than bitcoin. it is arguably more scalable, a key requirement pertaining to the long term affordability of a busy blockchain. because it is not focused on fulfilling financial transactions, the ethereum blockchain's purpose is different than bitcoin's. technically, ethereum doesn’t have a limit on the block size, and it will adjust dynamically as a whole, as part of its core design. in addition, ethereum is continuing to work on improving the scalability aspects, and that will have a direct benefit of lowering the overall transaction costs. in general, when looking at desirable features for a blockchain, the following characteristics come to mind, and they are ones that ethereum excels at: programmability scalability upgradability transactions manageability visibility affordability security speed/performance high availability extensibility third, whereas proof-of-work is ethereum’s current consensus process choice, it is planned to evolve into a less wasteful method known as the “proof-of-stake”. proof-of-stake has been proven to be an efficient and viable consensus method that is cheaper to operate while being more expensive to attack. finally, mining in the ethereum context can be done by regular computers, and doesn’t require the specialized computational horsepower that bitcoin requires, therefore ethereum's mining will be more affordable and accessible to the masses. anyone running the ethereum mining client software on their computer can become an ethereum miner, almost like bittorrent allows any user to openly share their media files. this is a good strategy, because it makes ethereum more affordable, by not being overly dependent on expensive mining. this also means that, unlike bitcoin; ethereum does not depend on accumulating a lot of mining power to work. it is more biased towards self-maintenance via a healthy balance of affordable mining and pay-per-play computing requirements. ethereum as freenet there is another interesting (and perhaps unexpected) ethereum analogy, and it is towards freenet. for background, freenet was an earlier way of making decentralized websites. it was free software that allowed users to anonymously share files, browse and publish "freesites" (web sites accessible only through freenet) and chat on forums, without fear of censorship, and with less vulnerability to attack. freenet’s philosophical underpinning was to enable and protect censorless freedom of speech, and to spread knowledge. freenet’s ideology had merits, but it also had two main flaws with known causes that ethereum remedies. first, downloading a website would often take more than one minute. second, it had no applications, and contained static content only. ethereum addresses these issues squarely via its native incentivization, and via its blockchain which is a backbone for enabling applications. more ethereum differentiation so, what are additional features of ethereum that differentiate it further and make it an attractive choice for developing decentralized applications? the development languages arguably, ethereum’s software development language capabilities are one of its biggest differentiating features, because programming the blockchain is a primordial objective. ethereum doesn't have one, but it has four such special-purpose languages: serpent (python inspired), solidity (javascript inspired), mutan (go inspired) and lll (lisp inspired). as one of the high-level programming languages used for ethereum, serpent (as suggested by its name) is designed to be very similar to python. it is intended to be maximally clean and simple, combining many of the efficiency benefits of a low-level language with ease-of-use in programming style. solidity is the preferred language, being designed from the ground-up for contract-oriented programming and the one under the most development. it inherits all the serpent qualities, but with a similar syntax to javascript which gives it a familiar adoption advantage that minimizes the barrier to entry, because javascript is commonly used by web developers. therefore, solidity leverages an existing skillset that millions of programmers already possess. another key feature of the ethereum blockchain is its “turing-completeness”, an important characteristic needed to ensure a high degree of universal solvability while running the required computation puzzles. more precisely, it is “quasi” turing-complete, because through setting up-front limits on the amount of computation allowed, it avoids the non-termination problem of a fully turing-complete language. furthermore, because ethereum’s languages are specifically designed for that blockchain, they provide incredible real-time granularity on transaction visibility and activity, a desirable feature that bitcoin has some challenges with. with bitcoin, you need to import the blockchain database, parse all transactions and query them in order to extract activity intelligence on it;  whereas with ethereum, you may issue specific address requests in real-time on the live blockchain. ethereum enables ðapps the available multiplicity of ethereum development languages is important because it enables developers to pick the ones they like in order to more easily and efficiently write decentralized applications (ðapps). a ðapp is the combined set of smart contracts and client-side code that enables them. smart contracts are like cryptographic "boxes" that contain a value and only unlock it if certain conditions are met. they typically encapsulate some logic, rules, a process, or an agreement between parties. when they are launched on ethereum, the network enforces their ramifications. a ðapp can be viewed architecturally as being very similar to a traditional web application, with one difference: in a traditional web application, you have client side javascript code that is run by users inside their browsers and server-side code that is run by a host or company; but in a ðapp, you instead have smart logic running on the blockchain, and client side code running in mist, the special browser. furthermore, a ðapp can interact or connect with other web apps or other decentralised technologies. for e.g., a ðapp may use a decentralized messaging service such as whisper in this ethereum’s context, or a decentralized file distribution like ipfs. on the web apps side, a company such as google might want to get data from a decentralized reputation service, or a bloomberg data feed may want to interface with a financial ðapp. the ethereum client ethereum includes a special purpose client browser that enables users to run the variety of available ðapps and launch smart contracts. this browser (called mist) is a user-friendly launching pad that promises to making smart contracts and ðapps widely accessible to a broad community of users. mist is ground-breaking in terms of lowering the user adoption barrier. its impact isn’t unlike what the web browser did for accessing the internet in a user-friendly manner, or what itunes represented for downloadable digital content. mist has special security layers, key management, decentralized account management (i.e. the user accounts are owned by the user and not a central third party authority), and includes blockchain-related components that will make it an indispensable tool for running or managing blockchain-specific decentralized applications for the average user who doesn’t need to understand technical aspects. from a user experience point of view, you use ðapps from inside of mist just as you interact with websites via a regular browser. for e.g., a pure ðapp such as augur (a prediction market) lives inside the ethereum mist browser. however, there can also exist services delivered through a regular browser in a more traditional web 2.0 manner that are still powered by and auditable through ethereum. the ethereum virtual machine when you think of these self-contained logic scripts that run on the blockchain, store data on it, and return some value(s) to the originator, it’s like running a program virtually in the cloud. simply said, these smart contracts are the code running on the ethereum virtual machine (evm). therefore, this is similar to a decentralized virtual computation service, except that it’s without the burden of web servers, and it is designed to be run by all participants in a peer-to-peer network that can read and write data and code to the blockchain safely and securely (via cryptographically secured digital signatures). the concept of the ethereum virtual machine (evm) is a very important one, because it is another primary innovation of the ethereum project. if you don't understand the evm, you don't understand ethereum. so, the evm "sits on the blockchain", but it’s actually made-up of several interconnected computers that anyone can upload programs to, leaving these programs to self-execute, and ensuring that current and all previous states of every program are always publicly visible. these programs run on the blockchain and continue to execute in exactly the way that the evm specifies. this allows anyone to create their own business logic for ownership, transaction formats and state transition functions. the ethereum core and its ecosystem at its inner core, ethereum is a multi-layered, cryptography-based, technical protocol that is open source. its various functional pieces are fully integrated by design, and the sum of its components turns it into a comprehensive platform for building and deploying modern decentralized applications. it was designed as a general-purpose decentralized application platform, with a full suite of tools that extend its functionality and power. although ethereum looks like a combination of several interconnected open source projects, its evolution has been clearly guided by a crystal vision to ensure that all the pieces fit together synergistically. like any major software platform, the ethereum core is augmented by a rich ecosystem comprised of a community of partners, technology interchange extensions, applications and ancillary services around it. more precisely, from a functional perspective, one could break up the ethereum ecosystem into three parts: core protocol technologies: p2p consensus, virtual machine, contracts, keys, blockchain, software languages and development environments, currency (fuel), technology integration and middleware services. applications: client software (mist or alethzero), mining, monitoring services, ðapps and other third party applications. ancillary services: mostly centered on education, research, learning and support via wikis, forums, the eth academy, web sites, bounty incentives, and a future developers conference. of particular interest is the applications segment whose size has already snowballed to over 100 various third-party projects, products, technology extensions and fully fledged businesses that are based on ethereum either entirely or partially. they include applications such as prediction markets, decentralized exchanges, crowdfunding, internet of things, voting and governance, gaming, reputation systems, social networking, chat messaging, insurance, healthcare, arts, ridesharing, distributed autonomous organizations, trading (financial instruments or commodities), accounting, communities, ecommerce, physical security, file storage, ownership stamping, content, microtransactions, community management, cloud computing, remittances, smart contracts management, smart assets, wallets, food, manufacturing, data storage, messaging, supply chain and others. (a future blog post will cover in more details the whole ethereum ecosystem, including the partner pieces). all these components point unequivocally to a healthy progression and growth of the ethereum footprint into a variety of financial and non-financial industry segments. ethereum’s programming capabilities offers vastly more power than that offered by bitcoin scripting because of the added powers of turing-completeness, value-awareness, blockchain-awareness and state transition logic capabilities. why should business people care? ethereum’s technology was built for developers primarily. but just like information technology, ethereum is an enabler of business opportunities, and as much as its initial support is vastly driven by a community of technologists, ethereum’s impact will get multiplied once business professionals also understand the potential of decentralizing applications, along with processes they enable. we need to learn how to apply what the blockchain technology gives us. ethereum developers and technology enthusiasts acknowledge that what they see is not enough in order to reveal the entire spectrum of the applications potential. developers need business people as partners who understand ethereum’s capabilities, and are willing to apply their own experience towards that understanding. during the 90's, the corporate reengineering trend caught on. it was driven by a desire to abolish old processes that offered little value, and replace them by simplified ones. suddenly, every large organization was reviewing their business processes, and identifying the ones that were slow, centralized, expensive, archaic, or did not meet users and customers needs. within the vision of ethereum, lies a reengineering promise parallel. the outcome would be to embark on a similar “crash and burn” exercise to find old applications that are slow, expensive, archaic or not meeting the required needs, and figure out if they could be re-thought via ðapps or truly decentralized applications. business people must learn about decentralization, and ask if old processes can be replaced by decentralized ones, enabled by ethereum. of specific interest are the types of applications that previously relied on central intermediaries or gatekeepers that held final authority on trust-related processes. ethereum enables the unbundling of trust, and repurposes it to run on the network. once you understand how to unbundle central processes, and relegate them to the edges of the network, you are only bound by the limits of your imagination. now is the time to shoot yourself in the foot, as it’s better than to have someone else shoot you in the head, tomorrow. progressive business and it managers should identify technical resources on their teams that need to learn about ethereum, and fund projects and implementations that promote innovation and reengineering efforts to discover and unlock the full potential of decentralized applications. if you fall in one of these categories, you are in an excellent position to start diving into ethereum: existing it departments startups with new ideas forward-thinking developers current bitcoin enthusiasts business innovators, leaders, investors and visionaries ngos, activists, researchers, and policy makers the best starting point for learning about developing on ethereum is the ethereum wiki. what can you do with ethereum? ethereum enables innovation via decentralized applications, equally touching industry, government and society segments. it is important to understand where decentralization and blockchains make the most sense, because we shouldn’t push those concepts on everything. from a technology point of view, decentralization is particularly valuable for "base layer" services, i.e. what everything else relies on, such as identity, reputation, communications, social networks, markets, and making them work without relying on specific trusted intermediaries in the middle. from a software applications perspective, blockchains are useful for enforcing decentralized consensus on databases that update themselves according to order-dependent state transition functions (where the speed of updates greater than one minute is acceptable), and this is where the network for powering ethereum-based smart contracts can shine. these can be used to securely execute a wide variety of services including: voting systems, domain name registries, financial exchanges, crowdfunding platforms, company governance, self-enforcing agreements, intellectual property, smart property, and distributed autonomous organizations. part of the ethereum vision is similar to bitcoin’s in terms of enabling decentralization, but the devil is in the implementation details. the ethereum platform has been designed from the ground-up to enable decentralized applications, whereas bitcoin’s genesis was a financial network that has been overloaded and overlaid with additional services and capabilities that are stretching its capabilities. protocols "built on top" of bitcoin are not light-client friendly, and the bitcoin blockchain is simply not designed to be scalable enough to support every transaction for every possible use case. developers are choosing ethereum because it’s easier to program advanced decentralized applications on ethereum than on bitcoin.  to think that bitcoin’s blockchain can do all of what ethereum is delivering is misinformed thinking. actually, a contrarian (but logical) view would be to think that bitcoin needs ethereum to succeed, because that increases the proof points and legitimacy for the viability of consensus-based applications development. ethereum developers and partners are solving and implementing big issues in this space, and their success will inspire the global crypto-technology ecosystem in its entirety. developers decide with their code. success will come based on the merits of each application or business initiative. whether an application runs on bitcoin or ethereum is important during the development phase, but once in the marketplace, users decide by adopting and engaging with the best applications. so, if ethereum development offers an advantage and you can exploit that, that's great. the critical masses we need are ones of users, developers, investors, and innovators, not just miners. in the long term, both bitcoin and ethereum are going to be viable platforms, each attracting the types of applications that best fit the respective platform characteristics. conclusion another way to view the cryptography field is in terms of bitcoin apps and blockchain apps. ethereum is squarely entrenched in the blockchain apps space which is a quite imaginative one. its superiority shines via the implementation of advanced decentralized applications whereas bitcoin’s blockchain has limitations, despite proposed overlay protocols, side chain projects and other bitcoin improvement proposals. to think that bitcoin is the only needed cryptocurrency/blockchain implies a lack of understanding for the field of cryptography-based computer science which is the larger theme at play. that would be like saying that fortran was the only language needed back in 1957 when it was deployed by ibm who developed it, just because it enjoyed some success then as the first industrial high-level programming language. or, if homes were just invented, saying that building homes with wood from trees is the only way to build them. and let’s not forget what ford said in 1922: “any customer can have a car painted any color that he wants so long as it is black.” many of the arguments for bitcoin dominance are biased by self-fulfilling prophecies or wishful scenarios. the cat is already out of the bag when it comes to innovation and adoption in alternative cryptocurrency protocols, and there is simply no real reason why bitcoin is destined to succeed on its own, and that other alternative protocols would fail. bitcoin and its blockchain don’t offer a silver bullet when it comes to decentralized applications development. there is hard work involved, but that work is a lot easier with ethereum. when you talk to developers, and see the level of activity around the ethereum technology, you will quickly realize that ethereum is spurring business and societal innovation at the global level, while opening the door to a new class of applications, never seen before. ethereum is leading the pack in decentralized applications protocols because of the support and vibrancy of its community members who are being drawn by the superiority of its platform capabilities. but ethereum is under no illusion that this won’t be a step-wise process, as it aims to gradually solve increasingly bigger problems, while starting with the more basic, simple and straight-forward ones. if bitcoin has shown us the future of money, ethereum is pointing to the future of decentralized software applications. if you are still not sure, ask your developer. william mougayar is a special advisor to eth dev’s executive directors. his work in technology spans 33 years as a 4x entrepreneur, book author, speaker, consultant, mentor and angel investor who previously held senior positions at hewlett-packard and cognizant. he blogs on startups and the cryptocurrency economy at startup management. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle a note on metcalfe's law, externalities and ecosystem splits 2017 jul 27 see all posts looks like it's blockchain split season again. for background of various people discussing the topic, and whether such splits are good or bad: power laws and network effects (arguing the btc/bcc split may destroy value due to network effect loss): https://medium.com/crypto-fundamental/power-laws-and-network-effects-why-bitcoincash-is-not-a-free-lunch-5adb579972aa brian armstrong on the ethereum hard fork (last year): https://blog.coinbase.com/on-the-ethereum-hard-fork-780f1577e986 phil daian on the eth/etc split: http://pdaian.com/blog/stop-worrying-love-etc/ given that ecosystem splits are not going away, and we may well see more of them in the crypto industry over the next decade, it seems useful to inform the discussion with some simple economic modeling. with that in mind, let's get right to it. suppose that there exist two projects a and b, and a set of users of total size \(n\), where a has \(n_a\) users and b has \(n_b\) users. both projects benefit from network effects, so they have a utility that increases with the number of users. however, users also have their own differing taste preferences, and this may lead them to choose the smaller platform over the bigger platform if it suits them better. we can model each individual's private utility in one of four ways: 1. \(u(a) = p + n_a\) \(u(b) = q + n_b\) 2. \(u(a) = p \cdot n_a\) \(u(b) = q \cdot n_b\) 3. \(u(a) = p + \ln{n_a}\) \(u(b) = q + \ln{n_b}\) 4. \(u(a) = p \cdot \ln{n_a}\)        \(u(b) = q \cdot \ln{n_b}\) \(p\) and \(q\) are private per-user parameters that you can think of as corresponding to users' distinct preferences. the difference between the first two approaches and the last two reflects differences between interpretations of metcalfe's law, or more broadly the idea that the per-user value of a system grows with the number of users. the original formulation suggested a per-user value of \(n\) (that is, a total network value of \(n^{2}\)), but other analysis (see here) suggests that above very small scales \(n\log{n}\) usually dominates; there is a controversy over which model is correct. the difference between the first and second (and between the third and fourth) is the extent to which utility from a system's intrinsic quality and utility from network effects are complementary that is, are the two things good in completely separate ways that do not interact with each other, like social media and coconuts, or are network effects an important part of letting the intrinsic quality of a system shine? we can now analyze each case in turn by looking at a situation where \(n_a\) users choose a and \(n_b\) users choose b, and analyze each individual's decision to choose one or the other from the perspective of economic externalities that is, does a user's choice to switch from a to b have a positive net effect on others' utility or a negative one? if switching has a positive externality, then it is virtuous and should be socially nudged or encouraged, and if it has a negative externality then it should be discouraged. we model an "ecosystem split" as a game where to start off \(n_a = n\) and \(n_b = 0\) and users are deciding for themselves whether or not to join the split, that is, to move from a to b, possibly causing \(n_a\) to fall and \(n_b\) to rise. switching (or not switching) from a to b has externalties because a and b both have network effects; switching from a to b has the negative externality of reducing a's network effect, and so hurting all remaining a users, but it also has the positive externality of increasing b's network effect, and so benefiting all b users. case 1 switching from a to b gives \(n_a\) users a negative externality of one, so a total loss of \(n_a\), and it gives \(n_b\) users a positive externality of one, so a total gain of \(n_b\). hence, the total externality is of size \(n_b n_a\); that is, switching from the smaller to the larger platform has positive externalities, and switching from the larger platform to the smaller platform has negative externalities. case 2 suppose \(p_a\) is the sum of \(p\) values of \(n_a\) users, and \(q_b\) is the sum of \(q\) values of \(n_b\) users. the total negative externality is \(p_a\) and the total positive externality is \(q_b\). hence, switching from the smaller platform to the larger has positive social externalities if the two platforms have equal intrinsic quality to their users (ie. users of a intrinsically enjoy a as much as users of b intrinsically enjoy b, so \(p\) and \(q\) values are evenly distributed), but if it is the case that a is bigger but b is better, then there are positive externalities in switching to b. furthermore, notice that if a user is making a switch from a larger a to a smaller b, then this itself is revealed-preference evidence that, for that user, and for all existing users of b, \(\frac{q}{p} > \frac{n_a}{n_b}\). however, if the split stays as a split, and does not proceed to become a full-scale migration, then that means that users of a hold different views, though this could be for two reasons: (i) they intrinsically dislike a but not by enough to justify the switch, (ii) they intrinsically like a more than b. this could arise because (a) a users have a higher opinion of a than b users, or (b) a users have a lower opinion of b than b users. in general, we see that moving from a system that makes its average user less happy to a system that makes its average user more happy has positive externalities, and in other situations it's difficult to say. case 3 the derivative of \(\ln{x}\) is \(\frac{1}{x}\). hence, switching from a to b gives \(n_a\) users a negative externality of \(\frac{1}{n_a}\), and it gives \(n_b\) users a positive externality of \(\frac{1}{n_b}\). hence, the negative and positive externalities are both of total size one, and thus cancel out. hence, switching from one platform to the other imposes no social externalities, and it is socially optimal if all users switch from a to b if and only if they think that it is a good idea for them personally to do so. case 4 let \(p_a\) and \(q_b\) are before. the negative externality is of total size \(\frac{p_a}{n_a}\) and the positive externality is of total size \(\frac{q_b}{n_b}\). hence, if the two systems have equal intrinsic quality, the externality is of size zero, but if one system has higher intrinsic quality, then it is virtuous to switch to it. note that as in case 2, if users are switching from a larger system to a smaller system, then that means that they find the smaller system to have higher intrinsic quality, although, also as in case 2, if the split remains a split and does not become a full-scale migration, then that means other users see the intrinsic quality of the larger system as higher, or at least not lower by enough to be worth the network effects. the existence of users switching to b suggests that for them, \(\frac{q}{p} \geq \frac{log{n_a}}{log{n_b}}\), so for the \(\frac{q_b}{n_b} > \frac{p_a}{n_a}\) condition to not hold (ie. for a move from a larger system to a smaller system not to have positive externalities) it would need to be the case that users of a have similarly high values of \(p\) an approximate heuristic is, the users of a would need to love a so much that if they were the ones in the minority that would be willing to split off and move to (or stay with) the smaller system. in general, it thus seems that moves from larger systems to smaller systems that actually do happen will have positive externalities, but it is far from ironclad that this is the case. hence, if the first model is true, then to maximize social welfare we should be trying to nudge people to switch to (or stay with) larger systems over smaller systems, and splits should be discouraged. if the fourth model is true, then we should be at least slightly trying to nudge people to switch to smaller systems over larger systems, and splits should be slightly encouraged. if the third model is true, then people will choose the socially optimal thing all by themselves, and if the second model is true, it's a toss-up. it is my personal view that the truth lies somewhere between the third and fourth models, and the first and second greatly overstate network effects above small scales. the first and second model (the \(n^{2}\) form of metcalfe's law) essentially state that a system growing from 990 million to 1 billion users gives the same increase in per-user utility as growing from 100,000 to 10.1 million users, which seems very unrealistic, whereas the \(n\log{n}\) model (growing from 100 million to 1 billion users gives the same increase in per-user utility as growing from 100,000 to 10 million users) intuitively seems much more correct. and the third model says: if you see people splitting off from a larger system to create a smaller system because they want something that more closely matches their personal values, then the fact that these people have already shown that they value this switch enough to give up the comforts of the original system's network effects is by itself enough evidence to show that the split is socially beneficial. hence, unless i can be convinced that the first model is true, or that the second model is true and the specific distributions of \(p\) and \(q\) values make splits make negative negative externalities, i maintain my existing view that those splits that actually do happen (though likely not hypothetical splits that end up not happening due to lack of interest) are in the long term socially beneficial, value-generating events. building the decentralized web 3.0 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search building the decentralized web 3.0 posted by taylor gerring on august 18, 2014 research & development how ethereum could shard the web given the state of our 25-year old web and all the problems inherited from legacy 1970's systems design, we should pause and take inventory of those components which are fundamentally broken and would offer a substantial return on development investment. intersecting this concern with security, privacy, and censorship resistance, it should be painfully obvious that an all-out attack on internet infrastructure is already underway. as netizens, a shared duty falls on us to explore, exploit, and implement new technologies that benefits creators, not oppressors. and while cryptography first allowed us to secure our messages from prying eyes, it is increasingly being used in more abstract ways like the secure movement of digital value via cryptocurrencies. if pgp was the first major iteration of applied crypto and bitcoin the second, then i anticipate that the interaction and integration of crypto into the very fabric of a decentralized web will be the refined third implementation, taking root and blossoming in popularity. the explosion of web services taking a look back at the brief history of the web, most would agree that web 1.0 was epitomized by cgi scripts generating templated content on a server and delivering it to the client in a final form. this was a clear model of monolithic centralization, however, this basic form of interactivity was a huge improvement over the basic post-and-read format that comprised much of internet content at that time. imagine having to reload the entire front page of digg every time you wanted to click something: digg in 2006, a prolific example of “web 2.0” interactivity not afforded by traditional cgi scripts as browser technology advanced, experimentation with ajax calls began, allowing us to asynchronously perform actions without having to reload the whole page. finally, you could upvote without submitting an html form and reloading everything. this movement to separate content from presentation—aided by css—pushed the web forward. today we have technologies like angularjs and emberjs which ask the designer to generate a client template with specific data holes to be filled in by some backend. although these frameworks facilitate some of the programming glue for seamless and live updates, they also nudge the developer to work in a specific way. but this is only a moderate step towards web 2.5. amuse-bouche the real web 3.0 has yet to begin, but it could obliterate the notion of separating content from presentation by removing the need to have servers at all. let's take a look at some of the underlying technologies the ethereum project aims to deliver: contracts: decentralized logic swarm: decentralized storage whisper: decentralized messaging interaction including ethereum contracts, swarm storage, whisper comms technologies like swarm could serve as the underlying static hosting infrastructure, removing the need to highly distribute and cache specific content. because “decentralized dropbox” has been discussed with such frequency, expect http-like bindings or services to be built atop this type of blob storage, making integration with the decentralized web 3.0 even simpler. this effort will also allow replacement of typical content delivery networks (cdn) with a distributed hash table (dht) pointing to file blobs, much how bittorrent works. because of the flexibility offered by ethereum contracts, the model of content access could be creator pays, reader pays, or some hybrid system. so we've just replaced the need to have caches, reverse proxies, cdns, load balancers, and the like to serve static content to users. another way in which etheruem could impact this traditional infrastructure is by replacing business logic application tiers with on-blockchain contracts. traditionally developed in a variety of web-friendly languages like perl, php, python, asp, c#, and ruby, ethereum contracts run in a fully-inspectable virtual machine that encourage simplicity and reuse. business analysts and project managers might find this code transparency refreshing, especially since the same code can be written in serpent (a python-like language), lll (a lisp-like language), xml (a nightmare), or even in visual block form! ethereum contract code visual editor how could all this be possible? taking a look at the latest ethereum proof-of-concept 6 javascript bindings, we see that a sprinkling of javascript is all that’s required to monitor an account balance on the decentralized web: <div>you have <span id="ether">?</span>.</div> <script> eth.watch({altered: eth.secrettoaddress(eth.key)}).changed(function() { document.getelementbyid("ether").innertext = eth.todecimal(eth.balanceat(eth.secrettoaddress(eth.key))) }); </script> because the ethereum protocol also acts as a large distributed key-store (a happy note for fans of nosql), eventually user accounts, credentials, and reputation can be migrated on-blockchain with the help of the whisper communication protocol. in this way, ethereum sets the stage for an total sharding of traditional infrastructure as we know it. no more complex high-availability infrastructure diagrams. in the ethereum ecosystem, even decentralized dns is free. your browser does not support the video tag. evaluating this context in a larger diagram of any systems infrastructure, it’s obvious that our current web isn't as privacy secure or censorship resistant as we desire. economies of scale have allowed single institutions to offer a vast amount of processing power and storage on the internet for very low prices, thereby increasing their market share to a point where they individually control large segments of internet activity, often under the supervision of less-than-savvy governments. in a post-borders era where the internet knows no bounds, such jurisdiction has little or no meaning. as economics of the ethereum ecosystem mature such that open contracts for lowest-rate storage develop, a free market of content hosting could evolve. given the nature and dynamics of p2p applications, popular content will readily scale as the swarm shares, rather than suffering from the buckling load of siloed servers. the net result is that popular content is delivered faster, not slower. we’ve spent decades optimizing the protocols that the internet was first founded on, but it’s time to recognize opportunities lost by continually patching the old system instead of curating a new, optimized one. the future will likely bring with it a transition period between traditional and decentralized technologies, where applications live in a hybrid universe and users are unaware of the turbulent undercurrent. but they should be. this metamorphosis will offer developers an opportunity to build the next-generation of decentralized, private, secure, censorship-resistant platforms that return control to creators and consumers of the next best idea. anyone with a dream is free to build on this new class of next-generation decentralized web services without owning a credit card or signing up for any accounts. although we are not told to or expected to, we have an imperative to cherish and improve the very shared resources that some wish to disturb, manipulate, and control. just as no single person fully understands the emerging internet collective intelligence, we should not expect any single entity to fully understand or maintain perfectly aligned motives. rather, we should rely on the internet to solve the problems of the internet. because of this, blockchain technologies like ethereum will allow for simplification and lowering of cost not seen since the introduction of infrastructure-as-a-service (iaas). extending the idea to beyond a simple web project, ethereum hopes to demonstrate how fully decentralized autonomous organizations (daos) can live wholly within cyberspace, negating not only the need for centralized servers, but also trusted third-parties, realizing the dreams of early internet pioneers that envisioned an independent new home of the mind. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements serpent upgrades: more fun stuff | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search serpent upgrades: more fun stuff posted by vitalik buterin on may 2, 2014 research & development over the past two weeks our primary focus has been getting all of the clients updated to poc5 compatibility, and it definitely has been a long road. among the changes to the vm include: the new init/code mechanism: basically, when you create a contract, the code provided will execute immediately, and then the return value of that code will be what becomes the contract’s code. this allows us to have contract initialization code, but still keep to the same format of [nonce, price, gas, to, value, data] for both transactions and contract creation, also making it easier to create new contracts via forwarding contracts reordering transaction and contract data: the order is now [nonce, price, gas, to, value, data] in transactions and [gas, to, value, datain, datainsz, dataout, dataoutsz] in messages. note that serpent retains the send(to, value, gas), o = msg(to, value, gas, datain, datainsz) and o = msg(to, value, gas, datain, datainsz, dataoutsz) parameters. fee adjustments: transaction creation now has a fee of 500 gas, and several other fees were updated. the codecopy and calldatacopy opcodes: codecopy takes code_index, mem_index, len as arguments, and copies the code from code_index ... code_index+len-1 to memory mem_index ... mem_index+len-1. these are very useful when combined with init/code. there is also now codesize. the largest changes, however, have been to the architecture surrounding the protocol. on the gui side, the c++ and go clients are evolving rapidly, and we will see more updates from that side coming very shortly. if you have been following ethereum closely, you have likely seen denny’s lotto, a full implementation of a lottery, plus gui, written and executed inside the c++ client. from here on, the c++ client will shift toward being a more developer-oriented tool, whereas the go client will start to focus on being a user-facing application (or rather, meta-application). on the compiler side, serpent has undergone a number of substantial improvements. first, the code. you can peek into the serpent compiler under the hood and you will be able to see all of the functionsavailable, together with their precise translations into evm code. for example, we have: 72:     ['access', 2, 1, 73:         ['', '', 32, 'mul', 'add', 'mload']], this means that what access(x,y) is actually doing under the hood is it’s recursively compiling whatever x and y actually are, and then loading the memory at index x + y * 32; hence, x is the pointer to the start of the array and y is the index. this code structure has been around since poc4, but now i have upgraded the meta-language used to describe translations even further, so as to include even if, while and init/code in this construction (before they were special cases); now, only set and seq remain as special cases, and if i wanted to i could even remove seq by reimplementing it as a rewrite rule. the largest changes so far have been for poc5 compatibility. for example, if you run serpent compile_to_assembly 'return(msg.data[0]*2)', you will see: ["begincode0.endcode0","dup","msize","swap","msize","begincode_0.endcode_0", "dup", "msize", "swap", "msize", "begincode0​.endcode0​","dup","msize","swap","msize","begincode_0", "calldatacopy", "return", "~begincode_0", "#code_begin", 2, 0, "calldataload", "mul", "msize", "swap", "msize", "mstore", 32, "swap", "return", "#code_end", "~endcode_0"] the actual code there is just: [2, 0, "calldataload", "mul", "msize", "swap", "msize", "mstore", 32, "swap", "return"] if you want to see what’s going on here, suppose that a message is coming in with its first datum being 5. we thus have: 2 -> stack: [2] 0 -> stack: [2, 0] calldataload -> stack: [2,5] mul -> stack: [10] msize -> stack: [10, 0] swap -> stack: [0, 10] msize -> stack: [0, 10, 0] mstore -> stack: [0], memory: [0, 0, 0 ... 10] 32 -> stack: [0, 32], memory: [0, 0, 0 ... 10] swap -> stack: [32, 0], memory: [0, 0, 0 ... 10] return the last return returns the 32 memory bytes starting from 0, or [0, 0, 0 ... 10], or the number 10. now, let’s analyze the wrapper code. ["begincode0.endcode0","dup","msize","swap","msize","begincode_0.endcode_0", "dup", "msize", "swap", "msize", "begincode0​.endcode0​","dup","msize","swap","msize","begincode_0", "calldatacopy", "return", "~begincode_0", "#code_begin", ..... , "#code_end", "~endcode_0"] i elided the inner code explained above to make things clearer. the first thing we see are two labels, begincode_0 andendcode_0, and the #code_begin and #code_end guards. the labels mark the beginning and end of the inner code, and the guards are there for the later stages of the compiler, which understands that everything between the guards should be compiled as if it is a separate program. now, let’s look at the first parts of the code. in this case, we have ~begincode_0 at position 10 and ~endcode_0 at position 24 in the final code. begincode0andbegincode_0 and begincode0​andendcode_0 are used to refer to these positions, and $begincode_0.endcode_0 refers to the length of the interval between them, 14. now, remember that during contract initialization the call data is the code that you’re feeding in. thus, we have: 14 -> stack: [14] dup -> stack: [14, 14] msize -> stack: [14, 14, 0] swap -> stack: [14, 0, 14] msize -> stack: [14, 0, 14, 0] 10 -> stack: [14, 0, 14, 0, 10] calldatacopy -> stack: [14, 0] memory: [ ... ] return notice how the first half of the code cleverly set up the stack so that it would push the inner code into memory indices 0…13, and then immediately return that chunk of memory. in the final compiled code,600e515b525b600a37f26002600035025b525b54602052f2, the inner code sits nicely to the right of the initializer code that simply returns it. in more complex contracts, initializers can also serve functions like setting certain storage slots to values, or even calling or creating other contracts. now, let us introduce the latest and most fun feature of serpent: imports. one common use case in contract land is that you want to give a contract the ability to spawn off new contracts. problem is, how to you put the code for the spawned contracts into the spawner contracts? before, the only solution was the uncomfortable approach of compiling the newer contracts first, and then putting the compiled code into an array. now, we have a better solution: import. put the following into returnten.se: x = create(tx.gas 100, 0, import(mul2.se)) return(msg(x,0,tx.gas-100,[5],1)) now, put the following into mul2.se: return(msg.data[0]*2) now, if you serpent compile returnten.se and run the contract, you notice that, voila, it returns ten. the reason why is obvious. the returnten.se contract creates an instance of the mul2.se contract, and then calls it with the value 5. mul2.se, as the name suggests, is a doubler, and so it returns 5*2 = 10. note that import is not a function in the standard sense; x = import('123.se') will fail, and import only works in the very specific context of create. now, suppose you are creating a 1000-line monster contract and want to split it up into files. to do that, we use inset. intoouter.se, put: if msg.data[0] == 1: inset(inner.se) and into inner.se, put: return(3) running serpent compile outer.se gives you a nice piece of compiled code that returns 3 if the msg.data[0] argument is equal to one. and that’s all there is to it. upcoming updates to serpent include: an improvement of this mechanism so it doesn’t load the inner code twice if you try to use import twice with the same filename string literals space and code-efficiency improvements for array literals a debugging decorator (ie. a compiling function which tells you what lines of serpent correspond to what bytes of compiled code) in the short term, though, my own effort will focus on bugfixes, a cross-client test suite, and continued work on ethereumjs-lib. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle a cbc casper tutorial 2018 dec 05 see all posts special thanks to vlad zamfir, aditya asgaonkar, ameen soleimani and jinglan wang for review in order to help more people understand "the other casper" (vlad zamfir's cbc casper), and specifically the instantiation that works best for blockchain protocols, i thought that i would write an explainer on it myself, from a less abstract and more "close to concrete usage" point of view. vlad's descriptions of cbc casper can be found here and here and here; you are welcome and encouraged to look through these materials as well. cbc casper is designed to be fundamentally very versatile and abstract, and come to consensus on pretty much any data structure; you can use cbc to decide whether to choose 0 or 1, you can make a simple block-by-block chain run on top of cbc, or a \(2^{92}\)-dimensional hypercube tangle dag, and pretty much anything in between. but for simplicity, we will first focus our attention on one concrete case: a simple chain-based structure. we will suppose that there is a fixed validator set consisting of \(n\) validators (a fancy word for "staking nodes"; we also assume that each node is staking the same amount of coins, cases where this is not true can be simulated by assigning some nodes multiple validator ids), time is broken up into ten-second slots, and validator \(k\) can create a block in slot \(k\), \(n + k\), \(2n + k\), etc. each block points to one specific parent block. clearly, if we wanted to make something maximally simple, we could just take this structure, impose a longest chain rule on top of it, and call it a day. the green chain is the longest chain (length 6) so it is considered to be the "canonical chain". however, what we care about here is adding some notion of "finality" the idea that some block can be so firmly established in the chain that it cannot be overtaken by a competing block unless a very large portion (eg. \(\frac{1}{4}\)) of validators commit a uniquely attributable fault act in some way which is clearly and cryptographically verifiably malicious. if a very large portion of validators do act maliciously to revert the block, proof of the misbehavior can be submitted to the chain to take away those validators' entire deposits, making the reversion of finality extremely expensive (think hundreds of millions of dollars). lmd ghost we will take this one step at a time. first, we replace the fork choice rule (the rule that chooses which chain among many possible choices is "the canonical chain", ie. the chain that users should care about), moving away from the simple longest-chain-rule and instead using "latest message driven ghost". to show how lmd ghost works, we will modify the above example. to make it more concrete, suppose the validator set has size 5, which we label \(a\), \(b\), \(c\), \(d\), \(e\), so validator \(a\) makes the blocks at slots 0 and 5, validator \(b\) at slots 1 and 6, etc. a client evaluating the lmd ghost fork choice rule cares only about the most recent (ie. highest-slot) message (ie. block) signed by each validator: latest messages in blue, slots from left to right (eg. \(a\)'s block on the left is at slot 0, etc.) now, we will use only these messages as source data for the "greedy heaviest observed subtree" (ghost) fork choice rule: start at the genesis block, then each time there is a fork choose the side where more of the latest messages support that block's subtree (ie. more of the latest messages support either that block or one of its descendants), and keep doing this until you reach a block with no children. we can compute for each block the subset of latest messages that support either the block or one of its descendants: now, to compute the head, we start at the beginning, and then at each fork pick the higher number: first, pick the bottom chain as it has 4 latest messages supporting it versus 1 for the single-block top chain, then at the next fork support the middle chain. the result is the same longest chain as before. indeed, in a well-running network (ie. the orphan rate is low), almost all of the time lmd ghost and the longest chain rule will give the exact same answer. but in more extreme circumstances, this is not always true. for example, consider the following chain, with a more substantial three-block fork: scoring blocks by chain length. if we follow the longest chain rule, the top chain is longer, so the top chain wins. scoring blocks by number of supporting latest messages and using the ghost rule (latest message from each validator shown in blue). the bottom chain has more recent support, so if we follow the lmd ghost rule the bottom chain wins, though it's not yet clear which of the three blocks takes precedence. the lmd ghost approach is advantageous in part because it is better at extracting information in conditions of high latency. if two validators create two blocks with the same parent, they should really be both counted as cooperating votes for the parent block, even though they are at the same time competing votes for themselves. the longest chain rule fails to capture this nuance; ghost-based rules do. detecting finality but the lmd ghost approach has another nice property: it's sticky. for example, suppose that for two rounds, \(\frac{4}{5}\) of validators voted for the same chain (we'll assume that the one of the five validators that did not, \(b\), is attacking): what would need to actually happen for the chain on top to become the canonical chain? four of five validators built on top of \(e\)'s first block, and all four recognized that \(e\) had a high score in the lmd fork choice. just by looking at the structure of the chain, we can know for a fact at least some of the messages that the validators must have seen at different times. here is what we know about the four validators' views: a's view c's view d's view e's view blocks produced by each validator in green, the latest messages we know that they saw from each of the other validators in blue. note that all four of the validators could have seen one or both of \(b\)'s blocks, and \(d\) and \(e\) could have seen \(c\)'s second block, making that the latest message in their views instead of \(c\)'s first block; however, the structure of the chain itself gives us no evidence that they actually did. fortunately, as we will see below, this ambiguity does not matter for us. \(a\)'s view contains four latest-messages supporting the bottom chain, and none supporting \(b\)'s block. hence, in (our simulation of) \(a\)'s eyes the score in favor of the bottom chain is at least 4-1. the views of \(c\), \(d\) and \(e\) paint a similar picture, with four latest-messages supporting the bottom chain. hence, all four of the validators are in a position where they cannot change their minds unless two other validators change their minds first to bring the score to 2-3 in favor of \(b\)'s block. note that our simulation of the validators' views is "out of date" in that, for example, it does not capture that \(d\) and \(e\) could have seen the more recent block by \(c\). however, this does not alter the calculation for the top vs bottom chain, because we can very generally say that any validator's new message will have the same opinion as their previous messages, unless two other validators have already switched sides first. a minimal viable attack. \(a\) and \(c\) illegally switch over to support \(b\)'s block (and can get penalized for this), giving it a 3-2 advantage, and at this point it becomes legal for \(d\) and \(e\) to also switch over. since fork choice rules such as lmd ghost are sticky in this way, and clients can detect when the fork choice rule is "stuck on" a particular block, we can use this as a way of achieving asynchronously safe consensus. safety oracles actually detecting all possible situations where the chain becomes stuck on some block (in cbc lingo, the block is "decided" or "safe") is very difficult, but we can come up with a set of heuristics ("safety oracles") which will help us detect some of the cases where this happens. the simplest of these is the clique oracle. if there exists some subset \(v\) of the validators making up portion \(p\) of the total validator set (with \(p > \frac{1}{2}\)) that all make blocks supporting some block \(b\) and then make another round of blocks still supporting \(b\) that references their first round of blocks, then we can reason as follows: because of the two rounds of messaging, we know that this subset \(v\) all (i) support \(b\) (ii) know that \(b\) is well-supported, and so none of them can legally switch over unless enough others switch over first. for some competing \(b'\) to beat out \(b\), the support such a \(b'\) can legally have is initially at most \(1-p\) (everyone not part of the clique), and to win the lmd ghost fork choice its support needs to get to \(\frac{1}{2}\), so at least \(\frac{1}{2} (1-p) = p \frac{1}{2}\) need to illegally switch over to get it to the point where the lmd ghost rule supports \(b'\). as a specific case, note that the \(p=\frac{3}{4}\) clique oracle offers a \(\frac{1}{4}\) level of safety, and a set of blocks satisfying the clique can (and in normal operation, will) be generated as long as \(\frac{3}{4}\) of nodes are online. hence, in a bft sense, the level of fault tolerance that can be reached using two-round clique oracles is \(\frac{1}{3}\), in terms of both liveness and safety. this approach to consensus has many nice benefits. first of all, the short-term chain selection algorithm, and the "finality algorithm", are not two awkwardly glued together distinct components, as they admittedly are in casper ffg; rather, they are both part of the same coherent whole. second, because safety detection is client-side, there is no need to choose any thresholds in-protocol; clients can decide for themselves what level of safety is sufficient to consider a block as finalized. going further cbc can be extended further in many ways. first, one can come up with other safety oracles; higher-round clique oracles can reach \(\frac{1}{3}\) fault tolerance. second, we can add validator rotation mechanisms. the simplest is to allow the validator set to change by a small percentage every time the \(q=\frac{3}{4}\) clique oracle is satisfied, but there are other things that we can do as well. third, we can go beyond chain-like structures, and instead look at structures that increase the density of messages per unit time, like the serenity beacon chain's attestation structure: in this case, it becomes worthwhile to separate attestations from blocks; a block is an object that actually grows the underlying dag, whereas an attestation contributes to the fork choice rule. in the serenity beacon chain spec, each block may have hundreds of attestations corresponding to it. however, regardless of which way you do it, the core logic of cbc casper remains the same. to make cbc casper's safety "cryptoeconomically enforceable", we need to add validity and slashing conditions. first, we'll start with the validity rule. a block contains both a parent block and a set of attestations that it knows about that are not yet part of the chain (similar to "uncles" in the current ethereum pow chain). for the block to be valid, the block's parent must be the result of executing the lmd ghost fork choice rule given the information included in the chain including in the block itself. dotted lines are uncle links, eg. when e creates a block, e notices that c is not yet part of the chain, and so includes a reference to c. we now can make cbc casper safe with only one slashing condition: you cannot make two attestations \(m_1\) and \(m_2\), unless either \(m_1\) is in the chain that \(m_2\) is attesting to or \(m_2\) is in the chain that \(m_2\) is attesting to. ok not ok the validity and slashing conditions are relatively easy to describe, though actually implementing them requires checking hash chains and executing fork choice rules in-consensus, so it is not nearly as simple as taking two messages and checking a couple of inequalities between the numbers that these messages commit to, as you can do in casper ffg for the no_surround and no_dbl_vote slashing conditions. liveness in cbc casper piggybacks off of the liveness of whatever the underlying chain algorithm is (eg. if it's one-block-per-slot, then it depends on a synchrony assumption that all nodes will see everything produced in slot \(n\) before the start of slot \(n+1\)). it's not possible to get "stuck" in such a way that one cannot make progress; it's possible to get to the point of finalizing new blocks from any situation, even one where there are attackers and/or network latency is higher than that required by the underlying chain algorithm. suppose that at some time \(t\), the network "calms down" and synchrony assumptions are once again satisfied. then, everyone will converge on the same view of the chain, with the same head \(h\). from there, validators will begin to sign messages supporting \(h\) or descendants of \(h\). from there, the chain can proceed smoothly, and will eventually satisfy a clique oracle, at which point \(h\) becomes finalized. chaotic network due to high latency. network latency subsides, a majority of validators see all of the same blocks or at least enough of them to get to the same head when executing the fork choice, and start building on the head, further reinforcing its advantage in the fork choice rule. chain proceeds "peacefully" at low latency. soon, a clique oracle will be satisfied. that's all there is to it! implementation-wise, cbc may arguably be considerably more complex than ffg, but in terms of ability to reason about the protocol, and the properties that it provides, it's surprisingly simple. dark mode toggle hard problems in cryptocurrency: five years later 2019 nov 22 see all posts special thanks to justin drake and jinglan wang for feedback in 2014, i made a post and a presentation with a list of hard problems in math, computer science and economics that i thought were important for the cryptocurrency space (as i then called it) to be able to reach maturity. in the last five years, much has changed. but exactly how much progress on what we thought then was important has been achieved? where have we succeeded, where have we failed, and where have we changed our minds about what is important? in this post, i'll go through the 16 problems from 2014 one by one, and see just where we are today on each one. at the end, i'll include my new picks for hard problems of 2019. the problems are broken down into three categories: (i) cryptographic, and hence expected to be solvable with purely mathematical techniques if they are to be solvable at all, (ii) consensus theory, largely improvements to proof of work and proof of stake, and (iii) economic, and hence having to do with creating structures involving incentives given to different participants, and often involving the application layer more than the protocol layer. we see significant progress in all categories, though some more than others. cryptographic problems blockchain scalability one of the largest problems facing the cryptocurrency space today is the issue of scalability ... the main concern with [oversized blockchains] is trust: if there are only a few entities capable of running full nodes, then those entities can conspire and agree to give themselves a large number of additional bitcoins, and there would be no way for other users to see for themselves that a block is invalid without processing an entire block themselves. problem: create a blockchain design that maintains bitcoin-like security guarantees, but where the maximum size of the most powerful node that needs to exist for the network to keep functioning is substantially sublinear in the number of transactions. status: great theoretical progress, pending more real-world evaluation. scalability is one technical problem that we have had a huge amount of progress on theoretically. five years ago, almost no one was thinking about sharding; now, sharding designs are commonplace. aside from ethereum 2.0, we have omniledger, lazyledger, zilliqa and research papers seemingly coming out every month. in my own view, further progress at this point is incremental. fundamentally, we already have a number of techniques that allow groups of validators to securely come to consensus on much more data than an individual validator can process, as well as techniques allow clients to indirectly verify the full validity and availability of blocks even under 51% attack conditions. these are probably the most important technologies: random sampling, allowing a small randomly selected committee to statistically stand in for the full validator set: https://github.com/ethereum/wiki/wiki/sharding-faq#how-can-we-solve-the-single-shard-takeover-attack-in-an-uncoordinated-majority-model fraud proofs, allowing individual nodes that learn of an error to broadcast its presence to everyone else: https://bitcoin.stackexchange.com/questions/49647/what-is-a-fraud-proof proofs of custody, allowing validators to probabilistically prove that they individually downloaded and verified some piece of data: https://ethresear.ch/t/1-bit-aggregation-friendly-custody-bonds/2236 data availability proofs, allowing clients to detect when the bodies of blocks that they have headers for are unavailable: https://arxiv.org/abs/1809.09044. see also the newer coded merkle trees proposal. there are also other smaller developments like cross-shard communication via receipts as well as "constant-factor" enhancements such as bls signature aggregation. that said, fully sharded blockchains have still not been seen in live operation (the partially sharded zilliqa has recently started running). on the theoretical side, there are mainly disputes about details remaining, along with challenges having to do with stability of sharded networking, developer experience and mitigating risks of centralization; fundamental technical possibility no longer seems in doubt. but the challenges that do remain are challenges that cannot be solved by just thinking about them; only developing the system and seeing ethereum 2.0 or some similar chain running live will suffice. timestamping problem: create a distributed incentive-compatible system, whether it is an overlay on top of a blockchain or its own blockchain, which maintains the current time to high accuracy. all legitimate users have clocks in a normal distribution around some "real" time with standard deviation 20 seconds ... no two nodes are more than 20 seconds apart the solution is allowed to rely on an existing concept of "n nodes"; this would in practice be enforced with proof-of-stake or non-sybil tokens (see #9). the system should continuously provide a time which is within 120s (or less if possible) of the internal clock of >99% of honestly participating nodes. external systems may end up relying on this system; hence, it should remain secure against attackers controlling < 25% of nodes regardless of incentives. status: some progress. ethereum has actually survived just fine with a 13-second block time and no particularly advanced timestamping technology; it uses a simple technique where a client does not accept a block whose stated timestamp is earlier than the client's local time. that said, this has not been tested under serious attacks. the recent network-adjusted timestamps proposal tries to improve on the status quo by allowing the client to determine the consensus on the time in the case where the client does not locally know the current time to high accuracy; this has not yet been tested. but in general, timestamping is not currently at the foreground of perceived research challenges; perhaps this will change once more proof of stake chains (including ethereum 2.0 but also others) come online as real live systems and we see what the issues are. arbitrary proof of computation problem: create programs poc_prove(p,i) -> (o,q) and poc_verify(p,o,q) -> { 0, 1 } such that poc_prove runs program p on input i and returns the program output o and a proof-of-computation q and poc_verify takes p, o and q and outputs whether or not q and o were legitimately produced by the poc_prove algorithm using p. status: great theoretical and practical progress. this is basically saying, build a snark (or stark, or shark, or...). and we've done it! snarks are now increasingly well understood, and are even already being used in multiple blockchains today (including tornado.cash on ethereum). and snarks are extremely useful, both as a privacy technology (see zcash and tornado.cash) and as a scalability technology (see zk rollup, starkdex and starking erasure coded data roots). there are still challenges with efficiency; making arithmetization-friendly hash functions (see here and here for bounties for breaking proposed candidates) is a big one, and efficiently proving random memory accesses is another. furthermore, there's the unsolved question of whether the o(n * log(n)) blowup in prover time is a fundamental limitation or if there is some way to make a succinct proof with only linear overhead as in bulletproofs (which unfortunately take linear time to verify). there are also ever-present risks that the existing schemes have bugs. in general, the problems are in the details rather than the fundamentals. code obfuscation the holy grail is to create an obfuscator o, such that given any program p the obfuscator can produce a second program o(p) = q such that p and q return the same output if given the same input and, importantly, q reveals no information whatsoever about the internals of p. one can hide inside of q a password, a secret encryption key, or one can simply use q to hide the proprietary workings of the algorithm itself. status: slow progress. in plain english, the problem is saying that we want to come up with a way to "encrypt" a program so that the encrypted program would still give the same outputs for the same inputs, but the "internals" of the program would be hidden. an example use case for obfuscation is a program containing a private key where the program only allows the private key to sign certain messages. a solution to code obfuscation would be very useful to blockchain protocols. the use cases are subtle, because one must deal with the possibility that an on-chain obfuscated program will be copied and run in an environment different from the chain itself, but there are many possibilities. one that personally interests me is the ability to remove the centralized operator from collusion-resistance gadgets by replacing the operator with an obfuscated program that contains some proof of work, making it very expensive to run more than once with different inputs as part of an attempt to determine individual participants' actions. unfortunately this continues to be a hard problem. there is continuing ongoing work in attacking the problem, one side making constructions (eg. this) that try to reduce the number of assumptions on mathematical objects that we do not know practically exist (eg. general cryptographic multilinear maps) and another side trying to make practical implementations of the desired mathematical objects. however, all of these paths are still quite far from creating something viable and known to be secure. see https://eprint.iacr.org/2019/463.pdf for a more general overview to the problem. hash-based cryptography problem: create a signature algorithm relying on no security assumption but the random oracle property of hashes that maintains 160 bits of security against classical computers (ie. 80 vs. quantum due to grover's algorithm) with optimal size and other properties. status: some progress. there have been two strands of progress on this since 2014. sphincs, a "stateless" (meaning, using it multiple times does not require remembering information like a nonce) signature scheme, was released soon after this "hard problems" list was published, and provides a purely hash-based signature scheme of size around 41 kb. additionally, starks have been developed, and one can create signatures of similar size based on them. the fact that not just signatures, but also general-purpose zero knowledge proofs, are possible with just hashes was definitely something i did not expect five years ago; i am very happy that this is the case. that said, size continues to be an issue, and ongoing progress (eg. see the very recent deep fri) is continuing to reduce the size of proofs, though it looks like further progress will be incremental. the main not-yet-solved problem with hash-based cryptography is aggregate signatures, similar to what bls aggregation makes possible. it's known that we can just make a stark over many lamport signatures, but this is inefficient; a more efficient scheme would be welcome. (in case you're wondering if hash-based public key encryption is possible, the answer is, no, you can't do anything with more than a quadratic attack cost) consensus theory problems asic-resistant proof of work one approach at solving the problem is creating a proof-of-work algorithm based on a type of computation that is very difficult to specialize ... for a more in-depth discussion on asic-resistant hardware, see https://blog.ethereum.org/2014/06/19/mining/. status: solved as far as we can. about six months after the "hard problems" list was posted, ethereum settled on its asic-resistant proof of work algorithm: ethash. ethash is known as a memory-hard algorithm. the theory is that random-access memory in regular computers is well-optimized already and hence difficult to improve on for specialized applications. ethash aims to achieve asic resistance by making memory access the dominant part of running the pow computation. ethash was not the first memory-hard algorithm, but it did add one innovation: it uses pseudorandom lookups over a two-level dag, allowing for two ways of evaluating the function. first, one could compute it quickly if one has the entire (~2 gb) dag; this is the memory-hard "fast path". second, one can compute it much more slowly (still fast enough to check a single provided solution quickly) if one only has the top level of the dag; this is used for block verification. ethash has proven remarkably successful at asic resistance; after three years and billions of dollars of block rewards, asics do exist but are at best 2-5 times more power and cost-efficient than gpus. progpow has been proposed as an alternative, but there is a growing consensus that asic-resistant algorithms will inevitably have a limited lifespan, and that asic resistance has downsides because it makes 51% attacks cheaper (eg. see the 51% attack on ethereum classic). i believe that pow algorithms that provide a medium level of asic resistance can be created, but such resistance is limited-term and both asic and non-asic pow have disadvantages; in the long term the better choice for blockchain consensus is proof of stake. useful proof of work making the proof of work function something which is simultaneously useful; a common candidate is something like folding@home, an existing program where users can download software onto their computers to simulate protein folding and provide researchers with a large supply of data to help them cure diseases. status: probably not feasible, with one exception. the challenge with useful proof of work is that a proof of work algorithm requires many properties: hard to compute easy to verify does not depend on large amounts of external data can be efficiently computed in small "bite-sized" chunks unfortunately, there are not many computations that are useful that preserve all of these properties, and most computations that do have all of those properties and are "useful" are only "useful" for far too short a time to build a cryptocurrency around them. however, there is one possible exception: zero-knowledge-proof generation. zero knowledge proofs of aspects of blockchain validity (eg. data availability roots for a simple example) are difficult to compute, and easy to verify. furthermore, they are durably difficult to compute; if proofs of "highly structured" computation become too easy, one can simply switch to verifying a blockchain's entire state transition, which becomes extremely expensive due to the need to model the virtual machine and random memory accesses. zero-knowledge proofs of blockchain validity provide great value to users of the blockchain, as they can substitute the need to verify the chain directly; coda is doing this already, albeit with a simplified blockchain design that is heavily optimized for provability. such proofs can significantly assist in improving the blockchain's safety and scalability. that said, the total amount of computation that realistically needs to be done is still much less than the amount that's currently done by proof of work miners, so this would at best be an add-on for proof of stake blockchains, not a full-on consensus algorithm. proof of stake another approach to solving the mining centralization problem is to abolish mining entirely, and move to some other mechanism for counting the weight of each node in the consensus. the most popular alternative under discussion to date is "proof of stake" that is to say, instead of treating the consensus model as "one unit of cpu power, one vote" it becomes "one currency unit, one vote". status: great theoretical progress, pending more real-world evaluation. near the end of 2014, it became clear to the proof of stake community that some form of "weak subjectivity" is unavoidable. to maintain economic security, nodes need to obtain a recent checkpoint extra-protocol when they sync for the first time, and again if they go offline for more than a few months. this was a difficult pill to swallow; many pow advocates still cling to pow precisely because in a pow chain the "head" of the chain can be discovered with the only data coming from a trusted source being the blockchain client software itself. pos advocates, however, were willing to swallow the pill, seeing the added trust requirements as not being large. from there the path to proof of stake through long-duration security deposits became clear. most interesting consensus algorithms today are fundamentally similar to pbft, but replace the fixed set of validators with a dynamic list that anyone can join by sending tokens into a system-level smart contract with time-locked withdrawals (eg. a withdrawal might in some cases take up to 4 months to complete). in many cases (including ethereum 2.0), these algorithms achieve "economic finality" by penalizing validators that are caught performing actions that violate the protocol in certain ways (see here for a philosophical view on what proof of stake accomplishes). as of today, we have (among many other algorithms): casper ffg: https://arxiv.org/abs/1710.09437 tendermint: https://tendermint.com/docs/spec/consensus/consensus.html hotstuff: https://arxiv.org/abs/1803.05069 casper cbc: ../../../2018/12/05/cbc_casper.html there continues to be ongoing refinement (eg. here and here) . eth2 phase 0, the chain that will implement ffg, is currently under implementation and enormous progress has been made. additionally, tendermint has been running, in the form of the cosmos chain for several months. remaining arguments about proof of stake, in my view, have to do with optimizing the economic incentives, and further formalizing the strategy for responding to 51% attacks. additionally, the casper cbc spec could still use concrete efficiency improvements. proof of storage a third approach to the problem is to use a scarce computational resource other than computational power or currency. in this regard, the two main alternatives that have been proposed are storage and bandwidth. there is no way in principle to provide an after-the-fact cryptographic proof that bandwidth was given or used, so proof of bandwidth should most accurately be considered a subset of social proof, discussed in later problems, but proof of storage is something that certainly can be done computationally. an advantage of proof-of-storage is that it is completely asic-resistant; the kind of storage that we have in hard drives is already close to optimal. status: a lot of theoretical progress, though still a lot to go, as well as more real-world evaluation. there are a number of blockchains planning to use proof of storage protocols, including chia and filecoin. that said, these algorithms have not been tested in the wild. my own main concern is centralization: will these algorithms actually be dominated by smaller users using spare storage capacity, or will they be dominated by large mining farms? economics stable-value cryptoassets one of the main problems with bitcoin is the issue of price volatility ... problem: construct a cryptographic asset with a stable price. status: some progress. <img src="../../../../images/progress-files/happy_face2.png" style="width:50px; height: 50px" ]class="transparent" /> makerdao is now live, and has been holding stable for nearly two years. it has survived a 93% drop in the value of its underlying collateral asset (eth), and there is now more than $100 million in dai issued. it has become a mainstay of the ethereum ecosystem, and many ethereum projects have or are integrating with it. other synthetic token projects, such as uma, are rapidly gaining steam as well. however, while the makerdao system has survived tough economic conditions in 2019, the conditions were by no means the toughest that could happen. in the past, bitcoin has fallen by 75% over the course of two days; the same may happen to ether or any other collateral asset some day. attacks on the underlying blockchain are an even larger untested risk, especially if compounded by price decreases at the same time. another major challenge, and arguably the larger one, is that the stability of makerdao-like systems is dependent on some underlying oracle scheme. different attempts at oracle systems do exist (see #16), but the jury is still out on how well they can hold up under large amounts of economic stress. so far, the collateral controlled by makerdao has been lower than the value of the mkr token; if this relationship reverses mkr holders may have a collective incentive to try to "loot" the makerdao system. there are ways to try to protect against such attacks, but they have not been tested in real life. decentralized public goods incentivization one of the challenges in economic systems in general is the problem of "public goods". for example, suppose that there is a scientific research project which will cost $1 million to complete, and it is known that if it is completed the resulting research will save one million people $5 each. in total, the social benefit is clear ... [but] from the point of view of each individual person contributing does not make sense ... so far, most problems to public goods have involved centralization additional assumptions and requirements: a fully trustworthy oracle exists for determining whether or not a certain public good task has been completed (in reality this is false, but this is the domain of another problem) status: some progress. the problem of funding public goods is generally understood to be split into two problems: the funding problem (where to get funding for public goods from) and the preference aggregation problem (how to determine what is a genuine public good, rather than some single individual's pet project, in the first place). this problem focuses specifically on the former, assuming the latter is solved (see the "decentralized contribution metrics" section below for work on that problem). in general, there haven't been large new breakthroughs here. there's two major categories of solutions. first, we can try to elicit individual contributions, giving people social rewards for doing so. my own proposal for charity through marginal price discrimination is one example of this; another is the anti-malaria donation badges on peepeth. second, we can collect funds from applications that have network effects. within blockchain land there are several options for doing this: issuing coins taking a portion of transaction fees at protocol level (eg. through eip 1559) taking a portion of transaction fees from some layer-2 application (eg. uniswap, or some scaling solution, or even state rent in an execution environment in ethereum 2.0) taking a portion of other kinds of fees (eg. ens registration) outside of blockchain land, this is just the age-old question of how to collect taxes if you're a government, and charge fees if you're a business or other organization. reputation systems problem: design a formalized reputation system, including a score rep(a,b) -> v where v is the reputation of b from the point of view of a, a mechanism for determining the probability that one party can be trusted by another, and a mechanism for updating the reputation given a record of a particular open or finalized interaction. status: slow progress. there hasn't really been much work on reputation systems since 2014. perhaps the best is the use of token curated registries to create curated lists of trustable entities/objects; the kleros erc20 tcr (yes, that's a token-curated registry of legitimate erc20 tokens) is one example, and there is even an alternative interface to uniswap (http://uniswap.ninja) that uses it as the backend to get the list of tokens and ticker symbols and logos from. reputation systems of the subjective variety have not really been tried, perhaps because there is just not enough information about the "social graph" of people's connections to each other that has already been published to chain in some form. if such information starts to exist for other reasons, then subjective reputation systems may become more popular. proof of excellence one interesting, and largely unexplored, solution to the problem of [token] distribution specifically (there are reasons why it cannot be so easily used for mining) is using tasks that are socially useful but require original human-driven creative effort and talent. for example, one can come up with a "proof of proof" currency that rewards players for coming up with mathematical proofs of certain theorems status: no progress, problem is largely forgotten. the main alternative approach to token distribution that has instead become popular is airdrops; typically, tokens are distributed at launch either proportionately to existing holdings of some other token, or based on some other metric (eg. as in the handshake airdrop). verifying human creativity directly has not really been attempted, and with recent progress on ai the problem of creating a task that only humans can do but computers can verify may well be too difficult. 15 [sic]. anti-sybil systems a problem that is somewhat related to the issue of a reputation system is the challenge of creating a "unique identity system" a system for generating tokens that prove that an identity is not part of a sybil attack ... however, we would like to have a system that has nicer and more egalitarian features than "one-dollar-one-vote"; arguably, one-person-one-vote would be ideal. status: some progress. there have been quite a few attempts at solving the unique-human problem. attempts that come to mind include (incomplete list!): humanitydao: https://www.humanitydao.org/ pseudonym parties: https://bford.info/pub/net/sybil.pdf poap ("proof of attendance protocol"): https://www.poap.xyz/ brightid: https://www.brightid.org/ with the growing interest in techniques like quadratic voting and quadratic funding, the need for some kind of human-based anti-sybil system continues to grow. hopefully, ongoing development of these techniques and new ones can come to meet it. 14 [sic]. decentralized contribution metrics incentivizing the production of public goods is, unfortunately, not the only problem that centralization solves. the other problem is determining, first, which public goods are worth producing in the first place and, second, determining to what extent a particular effort actually accomplished the production of the public good. this challenge deals with the latter issue. status: some progress, some change in focus. more recent work on determining value of public-good contributions does not try to separate determining tasks and determining quality of completion; the reason is that in practice the two are difficult to separate. work done by specific teams tends to be non-fungible and subjective enough that the most reasonable approach is to look at relevance of task and quality of performance as a single package, and use the same technique to evaluate both. fortunately, there has been great progress on this, particularly with the discovery of quadratic funding. quadratic funding is a mechanism where individuals can make donations to projects, and then based on the number of people who donated and how much they donated, a formula is used to calculate how much they would have donated if they were perfectly coordinated with each other (ie. took each other's interests into account and did not fall prey to the tragedy of the commons). the difference between amount would-have-donated and amount actually donated for any given project is given to that project as a subsidy from some central pool (see #11 for where the central pool funding could come from). note that this mechanism focuses on satisfying the values of some community, not on satisfying some given goal regardless of whether or not anyone cares about it. because of the complexity of values problem, this approach is likely to be much more robust to unknown unknowns. quadratic funding has even been tried in real life with considerable success in the recent gitcoin quadratic funding round. there has also been some incremental progress on improving quadratic funding and similar mechanisms; particularly, pairwise-bounded quadratic funding to mitigate collusion. there has also been work on specification and implementation of bribe-resistant voting technology, preventing users from proving to third parties who they voted for; this prevents many kinds of collusion and bribe attacks. decentralized success metrics problem: come up with and implement a decentralized method for measuring numerical real-world variables ... the system should be able to measure anything that humans can currently reach a rough consensus on (eg. price of an asset, temperature, global co2 concentration) status: some progress. this is now generally just called "the oracle problem". the largest known instance of a decentralized oracle running is augur, which has processed outcomes for millions of dollars of bets. token curated registries such as the kleros tcr for tokens are another example. however, these systems still have not seen a real-world test of the forking mechanism (search for "subjectivocracy" here) either due to a highly controversial question or due to an attempted 51% attack. there is also research on the oracle problem happening outside of the blockchain space in the form of the "peer prediction" literature; see here for a very recent advancement in the space. another looming challenge is that people want to rely on these systems to guide transfers of quantities of assets larger than the economic value of the system's native token. in these conditions, token holders in theory have the incentive to collude to give wrong answers to steal the funds. in such a case, the system would fork and the original system token would likely become valueless, but the original system token holders would still get away with the returns from whatever asset transfer they misdirected. stablecoins (see #10) are a particularly egregious case of this. one approach to solving this would be a system that assumes that altruistically honest data providers do exist, and creating a mechanism to identify them, and only allowing them to churn slowly so that if malicious ones start getting voted in the users of systems that rely on the oracle can first complete an orderly exit. in any case, more development of oracle tech is very much an important problem. new problems if i were to write the hard problems list again in 2019, some would be a continuation of the above problems, but there would be significant changes in emphasis, as well as significant new problems. here are a few picks: cryptographic obfuscation: same as #4 above ongoing work on post-quantum cryptography: both hash-based as well as based on post-quantum-secure "structured" mathematical objects, eg. elliptic curve isogenies, lattices... anti-collusion infrastructure: ongoing work and refinement of https://ethresear.ch/t/minimal-anti-collusion-infrastructure/5413, including adding privacy against the operator, adding multi-party computation in a maximally practical way, etc. oracles: same as #16 above, but removing the emphasis on "success metrics" and focusing on the general "get real-world data" problem unique-human identities (or, more realistically, semi-unique-human identities): same as what was written as #15 above, but with an emphasis on a less "absolute" solution: it should be much harder to get two identities than one, but making it impossible to get multiple identities is both impossible and potentially harmful even if we do succeed homomorphic encryption and multi-party computation: ongoing improvements are still required for practicality decentralized governance mechanisms: daos are cool, but current daos are still very primitive; we can do better fully formalizing responses to pos 51% attacks: ongoing work and refinement of https://ethresear.ch/t/responding-to-51-attacks-in-casper-ffg/6363 more sources of public goods funding: the ideal is to charge for congestible resources inside of systems that have network effects (eg. transaction fees), but doing so in decentralized systems requires public legitimacy; hence this is a social problem along with the technical one of finding possible sources reputation systems: same as #12 above in general, base-layer problems are slowly but surely decreasing, but application-layer problems are only just getting started. dark mode toggle a note on charity through marginal price discrimination 2017 mar 11 see all posts updated 2018-07-28. see end note. the following is an interesting idea that i had two years ago that i personally believe has promise and could be easily implemented in the context of a blockchain ecosystem, though if desired it could certainly also be implemented with more traditional technologies (blockchains would help get the scheme network effects by putting the core logic on a more neutral platform). suppose that you are a restaurant selling sandwiches, and you ordinarily sell sandwiches for $7.50. why did you choose to sell them for $7.50, and not $7.75 or $7.25? it clearly can't be the case that the cost of production is exactly $7.49999, as in that case you would be making no profit, and would not be able to cover fixed costs; hence, in most normal situations you would still be able to make some profit if you sold at $7.25 or $7.75, though less. why less at $7.25? because the price is lower. why less at $7.75? because you get fewer customers. it just so happens that $7.50 is the point at which the balance between those two factors is optimal for you. notice one consequence of this: if you make a slight distortion to the optimal price, then even compared to the magnitude of the distortion the losses that you face are minimal. if you raise prices by 1%, from $7.50 to $7.575, then your profit declines from $6750 to $6733.12 - a tiny 0.25% reduction. and that's profit - if you had instead donated 1% of the price of each sandwich, it would have reduced your profit by 5%. the smaller the distortion the more favorable the ratio: raising prices by 0.2% only cuts your profits down by 0.01%. now, you could argue that stores are not perfectly rational, and not perfectly informed, and so they may not actually be charging at optimal prices, all factors considered. however, if you don't know what direction the deviation is in for any given store, then even still, in expectation, the scheme works the same way - except instead of losing $17 it's more like flipping a coin where half the time you gain $50 and half the time you lose $84. furthermore, in the more complex scheme that we will describe later, we'll be adjusting prices in both directions simultaneously, and so there will not even be any extra risk no matter how correct or incorrect the original price was, the scheme will give you a predictable small net loss. also, the above example was one where marginal costs are high, and customers are picky about prices - in the above model, charging $9 would have netted you no customers at all. in a situation where marginal costs are much lower, and customers are less price-sensitive, the losses from raising or lowering prices would be even lower. so what is the point of all this? well, suppose that our sandwich shop changes its policy: it sells sandwiches for $7.55 to the general public, but lowers the prices to $7.35 for people who volunteered in some charity that maintains some local park (say, this is 25% of the population). the store's new profit is \(\$6682.5 \cdot 0.25+\$6742.5 \cdot 0.75=\$6727.5\) (that's a $22.5 loss), but the result is that you are now paying all 4500 of your customers 20 cents each to volunteer at that charity - an incentive size of $900 (if you just count the customers who actually do volunteer, $225). so the store loses a bit, but gets a huge amount of leverage, de-facto contributing at least $225 depending on how you measure it for a cost of only $22.5. now, what we can start to do is build up an ecosystem of "stickers", which are non-transferable digital "tokens" that organizations hand out to people who they think are contributing to worthy causes. tokens could be organized by category (eg. poverty relief, science research, environmental, local community projects, open source software development, writing good blogs), and merchants would be free to charge marginally lower prices to holders of the tokens that represent whatever causes they personally approve of. the next stage is to make the scheme recursive being or working for a merchant that offers lower prices to holders of green stickers is itself enough to merit you a green sticker, albeit one that is of lower potency and gives you a lower discount. this way, if an entire community approves of a particular cause, it may actually be profit-maximizing to start offering discounts for the associated sticker, and so economic and social pressure will maintain a certain level of spending and participation toward the cause in a stable equilibrium. as far as implementation goes, this requires: a standard for stickers, including wallets where people can hold stickers payment systems that have support for charging lower prices to sticker holders included at least a few sticker-issuing organizations (the lowest overhead is likely to be issuing stickers for charity donations, and for easily verifiable online content, eg. open source software and blogs) so this is something that can certainly be bootstrapped within a small community and user base and then let to grow over time. update 2017.03.14: here is an economic model/simulation showing the above implemented as a python script. update 2018.07.28: after discussions with others (glen weyl and several reddit commenters), i realized a few extra things about this mechanism, some encouraging and some worrying: the above mechanism could be used not just by charities, but also by centralized corporate actors. for example, a large corporation could offer a bribe of $40 to any store that offers the 20-cent discount to customers of its products, gaining additional revenue much higher than $40. so it's empowering but potentially dangerous in the wrong hands... (i have not researched it but i'm sure this kind of technique is used in various kinds of loyalty programs already) the above mechanism has the property that a merchant can "donate" \(\$x\) to charity at a cost of \(\$x^{2}\) (note: \(x^{2}<x\) at the scales we're talking about here). this gives it a structure that's economically optimal in certain ways (see quadratic voting), as a merchant that feels twice as strongly about some public good will be inclined to offer twice as large a subsidy, whereas most other social choice mechanisms tend to either undervalue (as in traditional voting) or overvalue (as in buying policies via lobbying) stronger vs weaker preferences. long-range attacks: the serious problem with adaptive proof of work | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search long-range attacks: the serious problem with adaptive proof of work posted by vitalik buterin on may 15, 2014 research & development our current proof of work design, blockchain-based proof of work, is the second iteration of our attempt to create a mining algorithm that is guaranteed to remain cpu-friendly and resistant to optimization by specialized hardware (asics) in the long term. our first attempt, dagger, tried to take the idea of memory-hard algorithms like scrypt one step further by creating an algorithm which is memory-hard to compute, but memory-easy to verify, using directed acyclic graphs (basically, trees where each node has multiple parents). our current strategy takes a much more rigorous track: make the proof of work involve executing random contracts from the blockchain. because the ethereum scripting language is turing-complete, an asic that can execute ethereum scripts is by definition an asic for general computation, ie. a cpu – a much more elegant argument than “this is memory-hard so you can’t parallelize as much”. of course, there are issues of “well, can you make specific optimizations and still get a large speedup”, but it can be argued that those are minor kinks to be worked out over time. the solution is also elegant because it is simultaneously an economic one: if someone does create an asic, then others will have the incentive to look for types of computation that the asic can’t do and “pollute” the blockchain with such contracts. unfortunately, however, there is one much larger obstacle to such schemes in general, and one which is unfortunately to some degree fundamental: long-range attacks. a long-range attack basically works as follows. in a traditional 51% attack, i put 100 bitcoins into a fresh new account, then send those 100 bitcoins to a merchant in exchange for some instant-delivery digital good (say, litecoins). i wait for delivery (eg. after 6 confirmations), but then i immediately start working on a new blockchain starting from one block before the transaction sending the 100 bitcoins, and put in a transaction instead sending those bitcoins back to myself. i then put more mining power into my fork than the rest of the network combined is putting into the main chain, and eventually my fork overtakes the main chain and thereby becomes the main chain, so at the end i have both the bitcoins and the litecoins. in a long-range attack, instead of starting a fork 6 blocks back, i start the fork 60000 blocks back, or even at the genesis block. in bitcoin, such a fork is useless, since you’re just increasing the amount of time you would need to catch up. in blockchain-based proof of work, however, it is a serious problem. the reason is that if you start a fork straight from the genesis block, then while your mining will be slow at first, after a few hundred blocks you will be able to fill the blockchain up with contracts that are very easy for you to mine, but difficult for everyone else. one example of such a contract is simply: i = 0 while sha3(i) != 0x8ff5b6afea3c68b6cd68bd429b9b64a708fa2273a93ea9f9e3c763257affee1f: i = i + 1 you know that the contract will take exactly one million rounds before the hash matches up, so you can calculate exactly how many steps and how much gas it will take to run and what the state will be at the end immediately, but other people will have no choice but to actually run through the code. an important property of such a scheme, a necessary consequence of the halting problem, is that it is actually impossible (as in, mathematically provably impossible, not hollywood impossible) to construct a mechanism for detecting such clever contracts in the general case without actually running them. hence, the long-range-attacker could fill the blockchain with such contracts, “mine” them, and convince the network that it is doing a massive amount of work when it is actually just taking the shortcut. thus, after a few days, our attacker will be “mining” billions of times faster than the main chain, and thereby quickly overtake it. notice that the above attack assumes little about how the algorithm actually works; all it assumes is that the condition for producing a valid block is dependent on the blockchain itself, and there is a wide range of variability in how much influence on the blockchain a single unit of computational power can have. one solution involves artificially capping the variability; this is done by requiring a tree-hashed computational stack trace alongside the contract algorithm, which is something that cannot be shortcut-generated because even if you know that the computation will terminate after 1 million steps and produce a certain output you still need to run those million steps yourself to produce all of the intermediate hashes. however, although this solves the long-range-attack problem it also ensures that the primary computation is not general computation, but rather computing lots and lots of sha3s – making the algorithm once again vulnerable to specialized hardware. proof of stake a version of this attack also exists for naively implemented proof of stake algorithms. in a naively implemented proof of stake, suppose that there is an attacker with 1% of all coins at or shortly after the genesis block. that attacker then starts their own chain, and starts mining it. although the attacker will find themselves selected for producing a block only 1% of the time, they can easily produce 100 times as many blocks, and simply create a longer blockchain in that way. originally, i thought that this problem was fundamental, but in reality it’s an issue that can be worked around. one solution, for example, is to note that every block must have a timestamp, and users reject chains with timestamps that are far ahead of their own. a long-range attack will thus have to fit into the same length of time, but because it involves a much smaller quantity of currency units its score will be much lower. another alternative is to require at least some percentage (say, 30%) of all coins to endorse either every block or every nth block, thereby absolutely preventing all attacks with less than that percent of coins. our own pos algorithm, slasher, can easily be retrofitted with either of these solutions. thus, in the long term, it seems like either pure proof of stake or hybrid pow/pos are the way that blockchains are going to go. in the case of a hybrid pow/pos, one can easily have a scheme where pos is used to resolve the issue described above with bbpow. what we’ll go with for ethereum 1.0 may be proof of stake, it might be a hybrid scheme, and it might be boring old sha3, with the understanding that asics will not be developed since manufacturers would see no benefit with the impending arrival of ethereum 2.0. however, there is still one challenge that arguably remains unresolved: the distribution model. for my own thoughts on that, stay tuned for the next part of this series. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements daos are not scary, part 1: self-enforcing contracts and factum law | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search daos are not scary, part 1: self-enforcing contracts and factum law posted by vitalik buterin on february 24, 2014 research & development many of the concepts that we promote over in ethereum land may seem incredibly futuristic, and perhaps even frightening, at times. we talk about so-called “smart contracts” that execute themselves without any need, or any opportunity, for human intervention or involvement, people forming skynet-like “decentralized autonomous organizations” that live entirely on the cloud and yet control powerful financial resources and can incentivize people to do very real things in the physical world, decentralized “math-based law”, and a seemingly utopian quest to create some kind of fully trust-free society. to the uninformed user, and especially to those who have not even heard of plain old bitcoin, it can be hard to see how these kinds of things are possible, and if they are why they can possibly be desirable. the purpose of this series will be to dissect these ideas in detail, and show exactly what we mean by each one, discussing its properties, advantages and limitations. the first installment of the series will talk about so-called “smart contracts”. smart contracts are an idea that has been around for several decades, but was given its current name and first substantially brought to the (cryptography-inclined) public’s attention by nick szabo in 2005. in essence, the definition of a smart contract is simple: a smart contract is a contract that enforces itself. that is to say, whereas a regular contract is a piece of paper (or more recently pdf document) containing text which implicitly asks for a judge to order a party to send money (or other property) to another party under certain conditions, a smart contract is a computer program that can be run on hardware which automatically executes those conditions. nick szabo uses the example of a vending machine: a canonical real-life example, which we might consider to be the primitive ancestor of smart contracts, is the humble vending machine. within a limited amount of potential loss (the amount in the till should be less than the cost of breaching the mechanism), the machine takes in coins, and via a simple mechanism, which makes a freshman computer science problem in design with finite automata, dispense change and product according to the displayed price. the vending machine is a contract with bearer: anybody with coins can participate in an exchange with the vendor. the lockbox and other security mechanisms protect the stored coins and contents from attackers, sufficiently to allow profitable deployment of vending machines in a wide variety of areas. smart contracts are the application of this concept to, well, lots of things. we can have smart financial contracts that automatically shuffle money around based on certain formulas and conditions, smart domain name sale orders that give the domain to whoever first sends in $200, perhaps even smart insurance contracts that control bank accounts and automatically pay out based on some trusted source (or combination of sources) supplying data about real-world events. smart property at this point, however, one obvious question arises: how are these contracts going to be enforced? just like traditional contracts, which are not worth the paper they’re written on unless there’s an actual judge backed by legal power enforcing them, smart contracts needs to be “plugged in” to some system in order to actually have power to do anything. the most obvious, and oldest, solution is hardware, an idea that also goes by the name “smart property”. nick szabo’s vending machine is the canonical example here. inside the vending machine, there is a sort of proto-smart-contract, containing a set of computer code that looks something like this: if button_pressed == "coca cola" and money_inserted >= 1.75: release("coca cola") return_change(money_inserted 1.75) else if button_pressed == "aquafina water" and money_inserted >= 1.25: release("aquafina water") return_change(money_inserted 1.25) else if ... the contract has four “hooks” into the outside world: the button_pressed and money_inserted variables as input, and therelease and return_change commands as output. all four of these depend on hardware, although we focus on the last three because human input is generally considered to be a trivial problem. if the contract was running on an android phone from 2007, it would be useless; the android phone has no way of knowing how much money was inserted into a slot, and certainly cannot release coca cola bottles or return change. on a vending machine, on the other hand, the contract carries some “force”, backed by the vending machine’s internal coca cola holdings and its physical security preventing people from just taking the coca cola without following the rules of the contract. another, more futuristic, application of smart property is rental cars: imagine a world where everyone has their own private key on a smartphone, and there is a car such that when you pay $100 to a certain address the car automatically starts responding commands signed by your private key for a day. the same principle can also be applied to houses. if that sounds far-fetched, keep in mind that office buildings are largely smart property already: access is controlled by access cards, and the question of which (if any) doors each card is valid for is determined by a piece of code linked to a database. and if the company has an hr system that automatically processes employment contracts and activates new employees access cards, then that employment contract is, to a slight extent, a smart contract. smart money and factum society however, physical property is very limited in what it can do. physical property has a limited amount of security, so you cannot practically do anything interesting with more than a few tens of thousands of dollars with a smart-property setup. and ultimately, the most interesting contracts involve transferring money. but how can we actually make that work? right now, we basically can’t. we can, theoretically, give contracts the login details to our bank accounts, and then have the contract send money under some conditions, but the problem is that this kind of contract is not really “self-enforcing”. the party making the contract can always simply turn the contract off just before payment is due, or drain their bank account, or even simply change the password to the account. ultimately, no matter how the contract is integrated into the system, someone has the ability to shut it off. how can we solve the problem? ultimately, the answer is one that is radical in the context of our wider society, but already very much old news in the world of bitcoin: we need a new kind of money. so far, the evolution of money has followed three stages: commodity money, commodity-backed money and fiat money. commodity money is simple: it’s money that is valuable because it is also simultaneously a commodity that has some “intrinsic” use value. silver and gold are perfect examples, and in more traditional societies we also have tea, salt (etymology note: this is where the word “salary” comes from), seashells and the like. next came commodity-backed money – banks issuing certificates that are valuable because they are redeemable for gold. finally, we have fiat money. the “fiat” in “fiat money” is just like in “fiat lux“, except instead of god saying “let there be light” it’s the federal government saying “let there be money”. the money has value largely because the government issuing it accepts that money, and only that money, as payment for taxes and fees, alongside several other legal privileges. with bitcoin, however, we have a new kind of money: factum money. the difference between fiat money and factum money is this: whereas fiat money is put into existence, and maintained, by a government (or, theoretically, some other kind of agency) producing it, factum money just is. factum money is simply a balance sheet, with a few rules on how that balance sheet can be updated, and that money is valid among that set of users which decides to accept it. bitcoin is the first example, but there are more. for example, one can have an alternative rule, which states that only bitcoins coming out of a certain “genesis transaction”, count as part of the balance sheet; this is called “colored coins”, and is also a kind of factum money (unless those colored coins are fiat or commodity-backed). the main promise of factum money, in fact, is precisely the fact that it meshes so well with smart contracts. the main problem with smart contracts is enforcement: if a contract says to send 200tobobifxhappens,andxdoeshappen,howdoweensurethat200 to bob if x happens, and x does happen, how do we ensure that 200tobobifxhappens,andxdoeshappen,howdoweensurethat200 actually gets sent to bob. the solution with factum money is incredibly elegant: the definition of the money, or more precisely the definition of the current balance sheet, is the result of executing all of the contracts. thus, if x does happen, then everyone will agree that bob has the extra $200, and if x does not happen then everyone will agree that bob has whatever bob had before. this is actually a much more revolutionary development than you might think at first; with factum money, we have created a way for contracts, and perhaps even law in general, to work, and be effective, without relying on any kind of mechanism whatsoever to enforce it. want a $100 fine for littering? then define a currency so that you have 100 units less if you litter, and convince people to accept it. now, that particular example is very far-fetched, and likely impractical without a few major caveats which we will discuss below, but it shows the general principle, and there are many more moderate examples of this kind of principle that definitely can be put to work. just how smart are smart contracts? smart contracts are obviously very effective for any kind of financial applications, or more generally any kind of swaps between two different factum assets. one example is a domain name sale; a domain, like google.com, is a factum asset, since it’s backed by a database on a server that only carries any weight because we accept it, and money can obviously be factum as well. right now, selling a domain is a complicated process that often requires specialized services; in the future, you may be able to package up a sale offer into a smart contract and put it on the blockchain, and if anyone takes it both sides of the trade will happen automatically – no possibility of fraud involved. going back to the world of currencies, decentralized exchange is another example, and we can also do financial contracts such as hedging and leverage trading. however, there are places where smart contracts are not so good. consider, for example, the case of an employment contract: a agrees to do a certain task for b in exchange for payment of x units of currency c. the payment part is easy to smart-contract-ify. however, there is a part that is not so easy: verifying that the work actually took place. if the work is in the physical world, this is pretty much impossible, since blockchains don’t have any way of accessing the physical world. even if it’s a website, there is still the question of assessing quality, and although computer programs can use machine learning algorithms to judge such characteristics quite effectively in certain cases, it is incredibly hard to do so in a public contract without opening the door for employees “gaming the system”. sometimes, a society ruled by algorithms is just not quite good enough. fortunately, there is a moderate solution that can capture the best of both worlds: judges. a judge in a regular court has essentially unlimited power to do what they want, and the process of judging does not have a particularly good interface; people need to file a suit, wait a significant length of time for a trial, and the judge eventually makes a decision which is enforced by the legal system – itself not a paragon of lightning-quick efficiency. private arbitration often manages to be cheaper and faster than courts, but even there the problems are still the same. judges in a factum world, on the other hand, are very much different. a smart contract for employment might look like this: if says(b,"a did the job") or says(j,"a did the job"): send(200, a) else if says(a,"a did not do the job") or says(j,"a did not do the job"): send(200, b) says is a signature verification algorithm; says(p,t) basically checks if someone had submitted a message with text t and a digital signature that verifies using p’s public key. so how does this contract work? first, the employer would send 200 currency units into the contract, where they would sit in escrow. in most cases, the employer and employee are honest, so either a quits and releases the funds back to b by signing a message saying “a did not do the job” or a does the job, b verifies that a did the job, and the contract releases the funds to a. however, if a does the job, and b disagrees, then it’s up to judge j to say that either a did the job or a did not do the job. note that j’s power is very carefully delineated; all that j has the right to do is say that either a did the job or a did not do the job. a more sophisticated contract might also give j the right to grant judgements within the range between the two extremes. j does not have the right to say that a actually deserves 600 currency units, or that by the way the entire relationship is illegal and j should get the 200 units, or anything else outside of the clearly defined boundaries. and j’s power is enforced by factum – the contract contains j’s public key, and thus the funds automatically go to a or b based on the boundaries. the contract can even require messages from 2 out of 3 judges, or it can have separate judges judge separate aspects of the work and have the contract automatically assign b’s work a quality score based on those ratings. any contract can simply plug in any judge in exactly the way that they want, whether to judge the truth or falsehood of a specific fact, provide a measurement of some variable, or be one of the parties facilitating the arrangement. how will this be better than the current system? in short, what this introduces is “judges as a service”. now, in order to become a “judge” you need to get hired at a private arbitration firm or a government court or start your own. in a cryptographically enabled factum law system, being a judge simply requires having a public key and a computer with internet access. as counterintuitive as it sounds, not all judges need to be well-versed in law. some judges can specialize in, for example, determining whether or not a product was shipped correctly (ideally, the postal system would do this). other judges can verify the completion of employment contracts. others would appraise damages for insurance contracts. it would be up to the contract writer to plug in judges of each type in the appropriate places in the contract, and the part of the contract that can be defined purely in computer code will be. and that’s all there is to it. the next part of this series will talk about the concept of trust, and what cryptographers and bitcoin advocates really mean when they talk about building a “trust-free” society. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle notes on blockchain governance 2017 dec 17 see all posts in which i argue that "tightly coupled" on-chain voting is overrated, the status quo of "informal governance" as practiced by bitcoin, bitcoin cash, ethereum, zcash and similar systems is much less bad than commonly thought, that people who think that the purpose of blockchains is to completely expunge soft mushy human intuitions and feelings in favor of completely algorithmic governance (emphasis on "completely") are absolutely crazy, and loosely coupled voting as done by carbonvotes and similar systems is underrated, as well as describe what framework should be used when thinking about blockchain governance in the first place. see also: https://medium.com/@vlad_zamfir/against-on-chain-governance-a4ceacd040ca one of the more interesting recent trends in blockchain governance is the resurgence of on-chain coin-holder voting as a multi-purpose decision mechanism. votes by coin holders are sometimes used in order to decide who operates the super-nodes that run a network (eg. dpos in eos, neo, lisk and other systems), sometimes to vote on protocol parameters (eg. the ethereum gas limit) and sometimes to vote on and directly implement protocol upgrades wholesale (eg. tezos). in all of these cases, the votes are automatic the protocol itself contains all of the logic needed to change the validator set or to update its own rules, and does this automatically in response to the result of votes. explicit on-chain governance is typically touted as having several major advantages. first, unlike the highly conservative philosophy espoused by bitcoin, it can evolve rapidly and accept needed technical improvements. second, by creating an explicit decentralized framework, it avoids the perceived pitfalls of informal governance, which is viewed to either be too unstable and prone to chain splits, or prone to becoming too de-facto centralized the latter being the same argument made in the famous 1972 essay "tyranny of structurelessness". quoting tezos documentation: while all blockchains offer financial incentives for maintaining consensus on their ledgers, no blockchain has a robust on-chain mechanism that seamlessly amends the rules governing its protocol and rewards protocol development. as a result, first-generation blockchains empower de facto, centralized core development teams or miners to formulate design choices. and: yes, but why would you want to make [a minority chain split] easier? splits destroy network effects. on-chain governance used to select validators also has the benefit that it allows for networks that impose high computational performance requirements on validators without introducing economic centralization risks and other traps of the kind that appear in public blockchains (eg. the validator's dilemma). so far, all in all, on-chain governance seems like a very good bargain.... so what's wrong with it? what is blockchain governance? to start off, we need to describe more clearly what the process of "blockchain governance" is. generally speaking, there are two informal models of governance, that i will call the "decision function" view of governance and the "coordination" view of governance. the decision function view treats governance as a function \(f(x_1, x_2 ... x_n) \rightarrow y\), where the inputs are the wishes of various legitimate stakeholders (senators, the president, property owners, shareholders, voters, etc) and the output is the decision. the decision function view is often useful as an approximation, but it clearly frays very easily around the edges: people often can and do break the law and get away with it, sometimes rules are ambiguous, and sometimes revolutions happen and all three of these possibilities are, at least sometimes, a good thing. and often even behavior inside the system is shaped by incentives created by the possibility of acting outside the system, and this once again is at least sometimes a good thing. the coordination model of governance, in contrast, sees governance as something that exists in layers. the bottom layer is, in the real world, the laws of physics themselves (as a geopolitical realist would say, guns and bombs), and in the blockchain space we can abstract a bit further and say that it is each individual's ability to run whatever software they want in their capacity as a user, miner, stakeholder, validator or whatever other kind of agent a blockchain protocol allows them to be. the bottom layer is always the ultimate deciding layer; if, for example, all bitcoin users wake up one day and decides to edit their clients' source code and replace the entire code with an ethereum client that listens to balances of a particular erc20 token contract, then that means that that erc20 token is bitcoin. the bottom layer's ultimate governing power cannot be stopped, but the actions that people take on this layer can be influenced by the layers above it. the second (and crucially important) layer is coordination institutions. the purpose of a coordination institution is to create focal points around how and when individuals should act in order to better coordinate behavior. there are many situations, both in blockchain governance and in real life, where if you act in a certain way alone, you are likely to get nowhere (or worse), but if everyone acts together a desired result can be achieved. an abstract coordination game. you benefit heavily from making the same move as everyone else. in these cases, it's in your interest to go if everyone else is going, and stop if everyone else is stopping. you can think of coordination institutions as putting up green or red flags in the air saying "go" or "stop", with an established culture that everyone watches these flags and (usually) does what they say. why do people have the incentive to follow these flags? because everyone else is already following these flags, and you have the incentive to do the same thing as what everyone else is doing. a byzantine general rallying his troops forward. the purpose of this isn't just to make the soldiers feel brave and excited, but also to reassure them that everyone else feels brave and excited and will charge forward as well, so an individual soldier is not just committing suicide by charging forward alone. strong claim: this concept of coordination flags encompasses all that we mean by "governance"; in scenarios where coordination games (or more generally, multi-equilibrium games) do not exist, the concept of governance is meaningless. in the real world, military orders from a general function as a flag, and in the blockchain world, the simplest example of such a flag is the mechanism that tells people whether or not a hard fork "is happening". coordination institutions can be very formal, or they can be informal, and often give suggestions that are ambiguous. flags would ideally always be either red or green, but sometimes a flag might be yellow, or even holographic, appearing green to some participants and yellow or red to others. sometimes that are also multiple flags that conflict with each other. the key questions of governance thus become: what should layer 1 be? that is, what features should be set up in the initial protocol itself, and how does this influence the ability to make formulaic (ie. decision-function-like) protocol changes, as well as the level of power of different kinds of agents to act in different ways? what should layer 2 be? that is, what coordination institutions should people be encouraged to care about? the role of coin voting ethereum also has a history with coin voting, including: dao proposal votes: https://daostats.github.io/proposals.html the dao carbonvote: http://v1.carbonvote.com/ the eip 186/649/669 carbonvote: http://carbonvote.com/ these three are all examples of loosely coupled coin voting, or coin voting as a layer 2 coordination institution. ethereum does not have any examples of tightly coupled coin voting (or, coin voting as a layer 1 in-protocol feature), though it does have an example of tightly coupled miner voting: miners' right to vote on the gas limit. clearly, tightly coupled voting and loosely coupled voting are competitors in the governance mechanism space, so it's worth dissecting: what are the advantages and disadvantages of each one? assuming zero transaction costs, and if used as a sole governance mechanism, the two are clearly equivalent. if a loosely coupled vote says that change x should be implemented, then that will serve as a "green flag" encouraging everyone to download the update; if a minority wants to rebel, they will simply not download the update. if a tightly coupled vote implements change x, then the change happens automatically, and if a minority wants to rebel they can install a hard fork update that cancels the change. however, there clearly are nonzero transaction costs associated with making a hard fork, and this leads to some very important differences. one very simple, and important, difference is that tightly coupled voting creates a default in favor of the blockchain adopting what the majority wants, requiring minorities to exert great effort to coordinate a hard fork to preserve a blockchain's existing properties, whereas loosely coupled voting is only a coordination tool, and still requires users to actually download and run the software that implements any given fork. but there are also many other differences. now, let us go through some arguments against voting, and dissect how each argument applies to voting as layer 1 and voting as layer 2. low voter participation one of the main criticisms of coin voting mechanisms so far is that, no matter where they are tried, they tend to have very low voter participation. the dao carbonvote only had a voter participation rate of 4.5%: additionally, wealth distribution is very unequal, and the results of these two factors together are best described by this image created by a critic of the dao fork: the eip 186 carbonvote had ~2.7 million eth voting. the dao proposal votes did not fare better, with participation never reaching 10%. and outside of ethereum things are not sunny either; even in bitshares, a system where the core social contract is designed around voting, the top delegate in an approval vote only got 17% of the vote, and in lisk it got up to 30%, though as we will discuss later these systems have other problems of their own. low voter participation means two things. first, the vote has a harder time achieving a perception of legitimacy, because it only reflects the views of a small percentage of people. second, an attacker with only a small percentage of all coins can sway the vote. these problems exist regardless of whether the vote is tightly coupled or loosely coupled. game-theoretic attacks aside from "the big hack" that received the bulk of the media attention, the dao also had a number of much smaller game-theoretic vulnerabilities; this article from hackingdistributed does a good job of summarizing them. but this is only the tip of the iceberg. even if all of the finer details of a voting mechanism are implemented correctly, voting mechanisms in general have a large flaw: in any vote, the probability that any given voter will have an impact on the result is tiny, and so the personal incentive that each voter has to vote correctly is almost insignificant. and if each person's size of the stake is small, their incentive to vote correctly is insignificant squared. hence, a relatively small bribe spread out across the participants may suffice to sway their decision, possibly in a way that they collectively might quite disapprove of. now you might say, people are not evil selfish profit-maximizers that will accept a $0.5 bribe to vote to give twenty million dollars to josh arza just because the above calculation says their individual chance of affecting anything is tiny; rather, they would altruistically refuse to do something that evil. there are two responses to this criticism. first, there are ways to make a "bribe" that are quite plausible; for example, an exchange can offer interest rates for deposits (or, even more ambiguously, use the exchange's own money to build a great interface and features), with the exchange operator using the large quantity of deposits to vote as they wish. exchanges profit from chaos, so their incentives are clearly quite misaligned with users and coin holders. second, and more damningly, in practice it seems like people, at least in their capacity as crypto token holders, are profit maximizers, and seem to see nothing evil or selfish about taking a bribe or two. as "exhibit a", we can look at the situation with lisk, where the delegate pool seems to have been successfully captured by two major "political parties" that explicitly bribe coin holders to vote for them, and also require each member in the pool to vote for all the others. here's liskelite, with 55 members (out of a total 101): here's liskgdt, with 33 members: and as "exhibit b" some voter bribes being paid out in ark: here, note that there is a key difference between tightly coupled and loosely coupled votes. in a loosely coupled vote, direct or indirect vote bribing is also possible, but if the community agrees that some given proposal or set of votes constitutes a game-theoretic attack, they can simply socially agree to ignore it. and in fact this has kind of already happened the carbonvote contains a blacklist of addresses corresponding to known exchange addresses, and votes from these addresses are not counted. in a tightly coupled vote, there is no way to create such a blacklist at protocol level, because agreeing who is part of the blacklist is itself a blockchain governance decision. but since the blacklist is part of a community-created voting tool that only indirectly influences protocol changes, voting tools that contain bad blacklists can simply be rejected by the community. it's worth noting that this section is not a prediction that all tightly coupled voting systems will quickly succumb to bribe attacks. it's entirely possible that many will survive for one simple reason: all of these projects have founders or foundations with large premines, and these act as large centralized actors that are interested in their platforms' success that are not vulnerable to bribes, and hold enough coins to outweigh most bribe attacks. however, this kind of centralized trust model, while arguably useful in some contexts in a project's early stages, is clearly one that is not sustainable in the long term. non-representativeness another important objection to voting is that coin holders are only one class of user, and may have interests that collide with those of other users. in the case of pure cryptocurrencies like bitcoin, store-of-value use ("hodling") and medium-of-exchange use ("buying coffees") are naturally in conflict, as the store-of-value prizes security much more than the medium-of-exchange use case, which more strongly values usability. with ethereum, the conflict is worse, as there are many people who use ethereum for reasons that have nothing to do with ether (see: cryptokitties), or even value-bearing digital assets in general (see: ens). additionally, even if coin holders are the only relevant class of user (one might imagine this to be the case in a cryptocurrency where there is an established social contract that its purpose is to be the next digital gold, and nothing else), there is still the challenge that a coin holder vote gives a much greater voice to wealthy coin holders than to everyone else, opening the door for centralization of holdings to lead to unencumbered centralization of decision making. or, in other words... and if you want to see a review of a project that seems to combine all of these disadvantages at the same time, see this: https://btcgeek.com/bitshares-trying-memorycoin-year-ago-disastrous-ends/. this criticism applies to both tightly coupled and loosely coupled voting equally; however, loosely coupled voting is more amenable to compromises that mitigate its unrepresentativeness, and we will discuss this more later. centralization let's look at the existing live experiment that we have in tightly coupled voting on ethereum, the gas limit. here's the gas limit evolution over the past two years: you might notice that the general feel of the curve is a bit like another chart that may be quite familiar to you: basically, they both look like magic numbers that are created and repeatedly renegotiated by a fairly centralized group of guys sitting together in a room. what's happening in the first case? miners are generally following the direction favored by the community, which is itself gauged via social consensus aids similar to those that drive hard forks (core developer support, reddit upvotes, etc; in ethereum, the gas limit has never gotten controversial enough to require anything as serious as a coin vote). hence, it is not at all clear that voting will be able to deliver results that are actually decentralized, if voters are not technically knowledgeable and simply defer to a single dominant tribe of experts. this criticism once again applies to tightly coupled and loosely coupled voting equally. update: since writing this, it seems like ethereum miners managed to up the gas limit from 6.7 million to 8 million all without even discussing it with the core developers or the ethereum foundation. so there is hope; but it takes a lot of hard community building and other grueling non-technical work to get to that point. digital constitutions one approach that has been suggested to mitigate the risk of runaway bad governance algorithms is "digital constitutions" that mathematically specify desired properties that the protocol should have, and require any new code changes to come with a computer-verifiable proof that they satisfy these properties. this seems like a good idea at first, but this too should, in my opinion, be viewed skeptically. in general, the idea of having norms about protocol properties, and having these norms serve the function of one of the coordination flags, is a very good one. this allows us to enshrine core properties of a protocol that we consider to be very important and valuable, and make them more difficult to change. however, this is exactly the sort of thing that should be enforced in loosely coupled (ie. layer two), rather than tightly coupled (layer one) form. basically any meaningful norm is actually quite hard to express in its entirety; this is part of the complexity of value problem. this is true even for something as seemingly unambiguous as the 21 million coin limit. sure, one can add a line of code saying assert total_supply <= 21000000, and put a comment around it saying "do not remove at all costs", but there are plenty of roundabout ways of doing the same thing. for example, one could imagine a soft fork that adds a mandatory transaction fee this is proportional to coin value * time since the coins were last sent, and this is equivalent to demurrage, which is equivalent to deflation. one could also implement another currency, called bjtcoin, with 21 million new units, and add a feature where if a bitcoin transaction is sent the miner can intercept it and claim the bitcoin, instead giving the recipient bjtcoin; this would rapidly force bitcoins and bjtcoins to be fungible with each other, increasing the "total supply" to 42 million without ever tripping up that line of code. "softer" norms like not interfering with application state are even harder to enforce. we want to be able to say that a protocol change that violates any of these guarantees should be viewed as illegitimate there should be a coordination institution that waves a red flag even if they get approved by a vote. we also want to be able to say that a protocol change that follows the letter of a norm, but blatantly violates its spirit, the protocol change should still be viewed as illegitimate. and having norms exist on layer 2 in the minds of humans in the community, rather than in the code of the protocol best achieves that goal. toward a balance however, i am also not willing to go the other way and say that coin voting, or other explicit on-chain voting-like schemes, have no place in governance whatsoever. the leading alternative seems to be core developer consensus, however the failure mode of a system being controlled by "ivory tower intellectuals" who care more about abstract philosophies and solutions that sound technically impressive over and above real day-to-day concerns like user experience and transaction fees is, in my view, also a real threat to be taken seriously. so how do we solve this conundrum? well, first, we can heed the words of slatestarcodex in the context of traditional politics: the rookie mistake is: you see that some system is partly moloch [ie. captured by misaligned special interests], so you say "okay, we'll fix that by putting it under the control of this other system. and we'll control this other system by writing ‘do not become moloch' on it in bright red marker." ("i see capitalism sometimes gets misaligned. let's fix it by putting it under control of the government. we'll control the government by having only virtuous people in high offices.") i'm not going to claim there's a great alternative, but the occasionally-adequate alternative is the neoliberal one – find a couple of elegant systems that all optimize along different criteria approximately aligned with human happiness, pit them off against each other in a structure of checks and balances, hope they screw up in different places like in that swiss cheese model, keep enough individual free choice around that people can exit any system that gets too terrible, and let cultural evolution do the rest. in blockchain governance, it seems like this is the only way forward as well. the approach for blockchain governance that i advocate is "multifactorial consensus", where different coordination flags and different mechanisms and groups are polled, and the ultimate decision depends on the collective result of all of these mechanisms together. these coordination flags may include: the roadmap (ie. the set of ideas broadcasted earlier on in the project's history about the direction the project would be going) consensus among the dominant core development teams coin holder votes user votes, through some kind of sybil-resistant polling system established norms (eg. non-interference with applications, the 21 million coin limit) i would argue that it is very useful for coin voting to be one of several coordination institutions deciding whether or not a given change gets implemented. it is an imperfect and unrepresentative signal, but it is a sybil-resistant one if you see 10 million eth voting for a given proposal, you cannot dismiss that by simply saying "oh, that's just hired russian trolls with fake social media accounts". it is also a signal that is sufficiently disjoint from the core development team that if needed it can serve as a check on it. however, as described above, there are very good reasons why it should not be the only coordination institution. and underpinnning it all is the key difference from traditional systems that makes blockchains interesting: the "layer 1" that underpins the whole system is the requirement for individual users to assent to any protocol changes, and their freedom, and credible threat, to "fork off" if someone attempts to force changes on them that they consider hostile (see also: http://vitalik.ca/general/2017/05/08/coordination_problems.html). tightly coupled voting is also okay to have in some limited contexts for example, despite its flaws, miners' ability to vote on the gas limit is a feature that has proven very beneficial on multiple occasions. the risk that miners will try to abuse their power may well be lower than the risk that any specific gas limit or block size limit hard-coded by the protocol on day 1 will end up leading to serious problems, and in that case letting miners vote on the gas limit is a good thing. however, "allowing miners or validators to vote on a few specific parameters that need to be rapidly changed from time to time" is a very far cry from giving them arbitrary control over protocol rules, or letting voting control validation, and these more expansive visions of on-chain governance have a much murkier potential, both in theory and in practice. crypto renaissance | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search crypto renaissance posted by mihai alisie on september 2, 2014 research & development “the best way to predict the future is to invent it.” alan kay january 3rd 2009 marked the beginning of a new era. the invention of the blockchain is a milestone in technology -the string of characters 36prz1khympqsyaqxsg8vwbuiq2eogxlo2 being just one of the side effects triggered by it. thanks to you and this technology, more than thirty one thousand bitcoins will go towards spawning an entire new digital realm. the unfettered peer to peer world wide digital finance grid is building itself out. you make this possible. more than 5000 years ago, the earliest economic systems were introduced in mesopotamia, setting the foundations of modern society. as a word, mesopotamia, originates from ancient greek and stands for "[land] between the rivers". as a civilization, one of their greatest achievements was the introduction of the so called cuneiform script. this marked the beginning of the information revolution within human society. unlike ethereum code, this invention consisted in wedge-shaped marks on clay tablets, made with a blunt reed. as rudimentary as this might sound today, this invention gave birth to concepts and solutions previously unheard of in a variety of areas such as accounting and legal contracts. the introduction of this new method for storing and distributing information and wealth made mesopotamia one of the largest and most thriving civilizations of its era. the unintended side-effects of this invention have led to our current information age society. before arriving to our modern information systems, our civilization went through three major recent information revolutions: printing press telecoms internet traces of these information revolutions can be seen everywhere from health to manufacturing sectors, all pointing towards a decentralized, interconnected future. now, the crypto information revolution is underway. the centralized economy paradigm society created thousands of years ago was a successful social experiment that has ultimately led humanity here, but now we have reached a point where we can cross the river. the currently available technologies provide us a new advanced toolkit designed for distributing and storing both information and wealth… at light speed. this is crypto renaissance in full swing. the (collaborative) information age “if you look at history, innovation doesn't come just from giving people incentives; it comes from creating environments where their ideas can connect.” steven johnson around the 14th century, the second huge information revolution was brewing thanks to johannes gutenberg’s invention of  the printing press. unlike its predecessors, this invention included moveable type faces enabling efficient and affordable printing production. by opening information dissemination on a larger scale and breadth than ever before, this invention fostered a new culture which created new opportunities for intellectual and societal growth. it appears that each time a new information technology becomes available, a number of patterns start to emerge. as a thought experiment, if we would overlap today’s technological renaissance with the 14th-17th century renaissance we would observe the following: introduced a new method for storing and distributing information in society period marked by important technical achievements allowing a faster propagation of ideas ignited experimentation and an intellectual revitalization triggered paradigm shifts deepening our understanding of the world, ourselves and the universe resulted in social, political and economic transformations if we draw the parallels we can see how information technologies enable positive growth within society since mesopotamian ages. it furthers progress in most fields by creating accessible mediums of information and that lead to a participatory culture with lesser or without social limitations. in the 20th and 21st century, humanity as a whole accelerated immensely. the first big step towards creating a new medium for information that propelled our society into the information age was the invention of the microchip. this technology made general computing possible. later on, the internet emerged as a global system of interconnected computer networks that use a standard protocol suite (tcp/ip) to link billions of devices worldwide. today, the internet is an international mesh network connecting more than two billion people allowing the free* flow of information, wealth and services to be exchanged without boundaries. this has set the stage for an explosion of interesting new concepts to materialize. the ripples caused by this information revolution can be felt throughout the fabric of our society as storms of bits eroding the pillars of a centralized paradigm. the internet is one of the most powerful social experiments ever invented. from a collaborative perspective, the internet and technology in general became a creation catalyst, exponentially increasing our collaborative potential. furthermore, the cost of coordination and collaboration was reduced drastically by making all participants with internet access universally available and connected. this allowed us as humans to transcend time and space limitations -keystroke by keystroke. emerging patterns: free culture and open movements twitter network vizualized “in real open source, you have the right to control your own destiny.” linus torvalds started around the 1970s, the earliest form of free software cultural movement has been ignited by richard stallman and his peers through their work tackling access and dissemination of information in this new technological realm. as digital freedom culture ideals spread across this new information grid the number of free software initiatives grew and evolved in different ways from encryption algorithms to operating systems to blockchain technologies. many established scholars of innovation did not anticipate the emergence of a distributed and open model for innovation that can aggressively compete with traditionally closed and proprietary models. looking back, who could have expected that swarms of online individuals acting without monetary incentive would build through distributed ad-hoc processes the largest encyclopedic body of knowledge in human history, or one of microsoft’s most aggressive competitors? in this line of thinking, open source can be seen as a pattern of collaboration in itself, being part of a bigger pattern: open collaboration. according to wikipedia, open collaboration is described as "any system of innovation or production that relies on goal-oriented yet loosely coordinated participants, who interact to create a product (or service) of economic value, which they make available to contributors and non-contributors alike". open collaboration appears as a pattern in many areas of our lives and represents one of the underlying principles behind many great achievements on the internet. wikipedia and linux are perfect examples of open cultures of knowledge advancing, edit by edit. their existence stands as a monument depicting the collaborative power of loosely associated groups, challenging  and transcending centralized architectures. open source continues to fascinate many economists, sociologists, scientists and many others thanks to its collaborative ability to defy economic dogmas based on top-down closed-control principles. these communities thrive thanks to decentralized problem solving, self-selected participation and self organization resulting in open collaboration. the emergence of these organizational models and sheer numbers of participants, in the hundreds of thousands, raised the question of motivation: “why do these people work and participate for 'free'?” the common view of purely self-interested participants, is clearly not the answer when many participate with no promise of a direct financial reward for their efforts. if writing code, designing decentralized software architectures, and solving tough cryptographic problems are construed by outsiders to be unremunerated blood, sweat, and tears, the contributors themselves are more likely to insist that the work is a source of significant satisfaction that derives from the pure joy of engagement in the work, or with the group or community, or both. the answer seems to lie in a more expansive view of ourselves as human beings that acknowledges, as well as the role of economic motivations, notions of enjoyment and having fun together with identity and the social benefits of community. challenge, enjoyment, and creativity are hallmarks of participation in this paradigm. through their work and actions these open source communities,  reveal macro homo reciprocan patterns. these patterns are valuable because they can offer insights into why things work this way or more precisely in this case “how do we work this way and what can we learn from it”. when you start considering the possibility that our species might actually be a collaborative rather than competitive one you're faced with a number of beliefs deeply encroached in our current society and world view. the current world wide accepted assumption is that our race appears to be composed of selfish individuals homo economicus, however this theory is challenged by billions of lines of running code and millions of individuals working together for free. ethereum project as a distributed innovation network “if you really want to innovate, it seems that decentralization generates experimentation. you want to discover what works and what doesn’t.” nicholas bloom open source communities represent the most radical edge of openness and sharing observed to date in complex technology development. the ethos of sharing and modification extends beyond code to the actual technology development process in the sense that community members engage in joint problem solving via open sharing of ideas and ongoing interaction. we can witness this behavior in our etherean community as well, with people swarming around common interests and values. adopting a monolith organizational structure was clearly not an optimal solution for this project as classic centralized organizations usually move and adapt, if they do, at glacier speed compared to these agile swarms of individuals. this is how our pursuit for a decentralized organizational structure, which creates favorable conditions for innovation and experimentation began. the core ideas behind distributed innovation appeared to fit best our needs. distributed innovation systems are designed from the ground up to lower the cost of participation for contributors. why is that important? firstly, because nurturing growth at the edges was always an important thought for us. and secondly, by reducing or eliminating barriers to entry you expand the population that can self-select into the community. the above diagram depicts the structure of the project in the context of multiple autonomous entities collaborating on different initiatives. in the collaboration process experimentation and co-experimentation are encouraged, hopefully sparking a wave of technologies and innovations that can be generally described as world positive. this creates a novel ecosystem for knowledge creation and presents both challenges and opportunities. on the upside it is compatible with the dao framework that we're seeking to implement in the near future. on the downside it's an experiment in itself might not go too well. in a true open source fashion,  feedback, suggestions and improvements are welcome. to summarize, many industries and institutions that are deeply entrenched within our society will most likely undergo significant changes in the coming years. most probably we will also undergo significant changes but if anything, when we look at the  the bigger picture, it appears that man's capacity for self-transformation is the only constant in our history. almost nothing has been invented yet [caption id="attachment_929" align="aligncenter" width="2000"] thomas kuhn used the duck-rabbit optical illusion to demonstrate the way in which a paradigm shift could cause one to see the same information in an entirely different way.[/caption] man on the train: hey, are you a dreamer? the dreamer: yeah. man on the train: haven't seen too many of you around lately. things have been tough lately for dreamers. they say dreaming is dead, no one does it anymore. it's not dead it's just that it's been forgotten, removed from our language. nobody teaches it so nobody knows it exists. and the dreamer is banished to obscurity. well, i'm trying to change all that, and i hope you are too. by dreaming, every day. dreaming with our hands and dreaming with our minds. our planet is facing the greatest problems it's ever faced, ever. so whatever you do, don't be bored. this is absolutely the most exciting time we could have possibly hoped to be alive. and things are just starting. -- waking life “everything that can be invented has been invented” is a quote attributed to former u.s. patent office director, charles h. duell. supposedly this was said during the 18 century when the traditional road to innovation was paved in secrecy and emphasized the accumulation of patents and intellectual property. today, thanks to countless open innovations we appear to be (very) far from having invented everything. we are only scratching the surface the more we invent, the more others can expand upon those ideas. we are standing on the shoulders of taller and taller giants accelerating in their growth. at the same time, during this period of rapid acceleration it is starting to become much clearer that if we are to survive as a species we need to learn to work together. the divide and conquer era is approaching its end. the united and empowered era is shining its light through millions of open minds co-architecting the next societal operating system. we learned from the first renaissance that technology enables a positive growth within society, through accessible mediums of information encouraging a participatory culture devoid of social limitations. the internet gave humanity a worldwide renaissance coffee shop where people with very different skills and areas of knowledge can bump into each other and exchange ideas. blockchain technologies offer us now a new medium for information storage and distribution. a fractal of opportunities is opening in front of our minds. the technology, knowledge and world-wide community support required to transcend century old paradigms moving closer towards a free, open and non-proprietary world are now within reach. this is definitely the most exciting time we could have hoped to be alive. and things are just starting. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements toward a 12-second block time | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search toward a 12-second block time posted by vitalik buterin on july 11, 2014 research & development one of the annoyances of the blockchain as a decentralized platform is the sheer length of delay before a transaction gets finalized. one confirmation in the bitcoin network takes ten minutes on average, but in reality due to statistical effects when one sends a transaction one can only expect a confirmation within ten minutes 63.2% of the time; 36.8% of the time it will take longer than ten minutes, 13.5% of the time longer than twenty minutes and 0.25% of the time longer than an hour. because of fine technical points involving finney attacks and sub-50% double spends, for many use cases even one confirmation is not enough; gambling sites and exchanges often need to wait for three to six blocks to appear, often taking over an hour, before a deposit is confirmed. in the time before a transaction gets into a block, security is close to zero; although many miners refuse to forward along transactions that conflict with transactions that had already been sent earlier, there is no economic necessity for them to do so (in fact quite the contrary), and some don't, so reversing an unconfirmed transaction is possible with about a 10-20% success rate. in many cases, this is fine; if you pay for a laptop online, and then manage to yank back the funds five minutes later, the merchant can simply cancel the shipping; online subscription services work the same way. however, in the context of some in-person purchases and digital goods purchases, it is highly inconvenient. in the case of ethereum, the inconvenience is greater; we are trying to be not just a currency, but rather a generalized platform for decentralized applications, and especially in the context of non-financial apps people tend to expect a much more rapid response time. thus, for our purposes, having a blockchain that is faster than 10 minutes is critical. however, the question is, how low can we go, and if we go too low does that destabilize anything? overview of mining first off, let us have a quick overview of how mining works. the bitcoin blockchain is a series of blocks, with each one pointing to (ie. containing the hash of) the previous. each miner in the network attempts to produce blocks by first grabbing up the necessary data (previous block, transactions, time, etc), building up the block header, and then continually changing a value called the nonce until the nonce satisfies a function called a "proof of work condition" (or "mining algorithm"). this algorithm is random and usually fails; on average, in bitcoin the network needs to collectively make about 1020 attempts before a valid block is found. once some random miner finds a block that is valid (ie. it points to a valid previous block, its transactions and metadata are valid, and its nonce satisfies the pow condition), then that block is broadcast to the network and the cycle begins again. as a reward, the miner of that block gets some quantity of coins (25 btc in bitcoin) as a reward. the "score" of a block is defined in a simplified model as the number of blocks in the chain going back from it all the way to the genesis (formally, it's the total mining difficulty, so if the difficulty of the proof of work condition increases blocks created under this new more stringent condition count for more). the block that has the highest score is taken to be "truth". a subtle, but important, point is that in this model the incentive for miners is always to mine on the block with the highest score, because the block with the highest score is what users ultimately care about, and there are never any factors that make a lower-score block better. if we fool around with the scoring model, then if we are not careful this might change; but more on this later. we can model this kind of network thus: however, the problems arise when we take into account the fact that network propagation is not instant. according to a 2013 paper from decker and wattenhofer in zurich, once a miner produces a block on average it takes 6.5 seconds for the block to reach 50% of nodes, 40 seconds for it to reach 95% of nodes and the mean delay is 12.6 seconds. thus, a more accurate model might be: this gives rise to the following problem: if, at time t = 500, miner m mines a block b' on top of b (where "on top of" is understood to mean "pointing to as the previous block in the chain"), then miner n might not hear about the block until time t = 510, so until t = 510 miner n will still be mining on b. if miner b finds a block in that interval, then the rest of the network will reject miner b's block because they already saw miner m's block which has an equal score: stales, efficiency and centralization so what's wrong with this? actually, two things. first, it weakens the absolute strength of the network against attacks. at a block time of 600 seconds, as in bitcoin, this is not an issue; 12 seconds is a very small amount of time, and decker and wattenhofer estimate the total stale rate as being around 1.7%. hence, an attacker does not actually need 50.001% of the network in order to launch a 51% attack; if the attacker is a single node, they would only need 0.983 / 1 + 0.983 = 49.5%. we can estimate this via a mathematical formula: if transit time is 12 seconds, then after a block is produced the network will be producing stales for 12 seconds before the block propagates, so we can assume an average of 12 / 600 = 0.02 stales per valid block or a stale rate of 1.97%. at 60 seconds per block, however, we get 12 / 60 = 0.2 stales per valid block or a stale rate of 16.67%. at 12 seconds per block, we get 12 / 12 = 1 stale per valid block, or a stale rate of 50%. thus, we can see the network get substantially weaker against attacks. however, there is also another negative consequence of stale rates. one of the more pressing issues in the mining ecosystem is the problem of mining centralization. currently, most of the bitcoin network is split up into a small number of "mining pools", centralized constructions where miners share resources in order to receive a more even reward, and the largest of these pools has for months been bouncing between 33% and 51% of network hashpower. in the future, even individual miners may prove threatening; right now 25% of all new bitcoin mining devices are coming out of a single factory in shenzhen, and if the pessimistic version of my economic analysis proves correct that may eventually morph into 25% of all bitcoin miners being in a single factory in shenzhen. so how do stale rates affect centralization? the answer is a clever one. suppose that you have a network with 7000 pools with 0.01% hashpower, and one pool with 30% hashpower. 70% of the time, the last block is produced by one of these miners, and the network hears about it in 12 seconds, and things are somewhat inefficient but nevertheless fair. 30% of the time, however, it is the 30% hashpower mining pool that produced the last block; thus, it "hears" about the block instantly and has a 0% stale rate, whereas everyone else still has their full stale rate. because our model is still pretty simple, we can still do some math on an approximation in closed form. assuming a 12 second transit time and a 60-second block time, we have a stale rate of 16.67% as described above. the 30% mining pool will have a 0% stale rate 30% of the time, so its efficiency multiplier will be 0.833 * 0.7 + 1 * 0.3 = 0.8831, whereas everyone else will have an efficiency multiplier of 0.833; that's a 5.7% efficiency gain which is pretty economically significant especially for mining pools where the difference in fees is only a few percent either way. thus, if we want a 60 second block time, we need a better strategy. ghost the beginnings of a better approach come from a paper entitled "fast money grows on trees, not chains", published by aviv zohar and yonatan sompolinsky in december 2013. the idea is that even though stale blocks are not currently counted as part of the total weight of the chain, they could be; hence they propose a blockchain scoring system which takes stale blocks into account even if they are not part of the main chain. as a result, even if the main chain is only 50% efficient or even 5% efficient, an attacker attempting to pull off a 51% attack would still need to overcome the weight of the entire network. this, theoretically, solves the efficiency issue all the way down to 1-second block times. however, there is a problem: the protocol, as described, only includes stales in the scoring of a blockchain; it does not assign the stales a block reward. hence, it does nothing to solve the centralization problem; in fact, with a 1-second block time the most likely scenario involves the 30% mining pool simply producing every block. of course, the 30% mining pool producing every block on the main chain is fine, but only if the blocks off chain are also fairly rewarded, so the 30% mining pool still collects not much more than 30% of the revenue. but for that rewarding stales will be required. now, we can't reward all stales always and forever; that would be a bookkeeping nightmare (the algorithm would need to check very diligently that a newly included uncle had never been included before, so we would need an "uncle tree" in each block alongside the transaction tree and state tree) and more importantly it would make double-spends cost-free. thus, let us construct our first protocol, single-level ghost, which does the minimal thing and takes uncles only up to one level (this is the algorithm used in ethereum up to now): every block must point to a parent (ie. previous block), and can also include zero or more uncles. an "uncle" is defined as a block with a valid header (the block itself need not be valid, since we only care about its proof-of-work) which is the child of the parent of the parent of the block but not the parent (ie. the standard definition of "uncle" from genealogy that you learned at age 4). a block on the main chain gets a reward of 1. when a block includes an uncle, the uncle gets a reward of 7/8 and the block including the uncle gets a reward of 1/16. the score of a block is zero for the genesis block, otherwise the score of the parent plus the difficulty of the block multiplied by one plus the number of included uncles. thus, in the graphical blockchain example given above, we'll instead have something like this: here, the math gets more complex, so we'll make some intuitive arguments and then take the lazy approach and simulate the whole thing. the basic intuitive argument is this: in the basic mining protocol, for the reasons we described above, the stale rate is roughly t/(t+t) where t is the transit time and t is the block interval, because t/t of the time miners are mining on old data. with single-level ghost, the failure condition changes from mining one stale to mining two stales in a row (since uncles can get included but relatives with a divergence of 2 or higher cannot), so the stale rate should be (t/t)^2, ie. about 2.7% instead of 16.7%. now, let's use a python script to test that theory: ### printing results ### 1 1.0 10 10.2268527074 25 25.3904084273 5 4.93500893242 15 14.5675475882 total blocks produced: 16687 total blocks in chain: 16350 efficiency: 0.979804638341 average uncles: 0.1584242596 length of chain: 14114 block time: 70.8516366728 the results can be parsed as follows. the top five numbers are a centralization indicator; here, we see that a miner with 25% hashpower gets 25.39x as much reward as a miner with 1% hashpower. the efficiency is 0.9798 meaning that 2.02% of all blocks are not included at all, and there are 0.158 uncles per block; hence, our intuitions about a ~16% stale rate without uncle inclusion and 2.7% with uncle inclusion are confirmed almost exactly. note that the actual block time is 70.85s because even though there is a valid proof of work solution every 60s, 2% of them are lost and 14% of them make it into only the next block as an uncle, not into the main chain. now, there is a problem here. the original authors of the ghost paper did not include uncle/stale rewards, and although i believe it is a good idea to deviate from their prescription for the reasons i described above, they did not do so for a reason: it makes the economic analysis more uncomfortable. specifically, when only the main chain gets rewarded there is an unambiguous argument why it's always worth it to mine on the head and not some previous block, namely the fact that the only thing that conceivably differentiates any two blocks is their score and higher score is obviously better than lower score, but once uncle rewards are introduced there are other factors that make things somewhat tricky. specifically, suppose that the main chain has its last block m (score 502) with parent l (score 501) with parent k (score 500). also suppose that k has two stale children, both of which were produced after m so there was no chance for them to be included in m as uncles. if you mine on m, you would produce a block with score 502 + 1 = 503 and reward 1, but if you mine on l you would be able to include k's children and get a block with score 501 + 1 + 2 = 504 and reward 1 + 0.0625 * 2 = 1.125. additionally, there is a selfish-mining-esque attack against single-level ghost. the argument is as follows: if a mining pool with 25% hashpower were not to include any other blocks, then in the short term it would hurt itself because it would no longer receive the 1/16x nephew reward but it would hurt others more. because in the long-term mining is a zero-sum game since the block time rebalances to keep issuance constant, this means that not including uncles might actually be a dominant strategy, so centralization concerns are not entirely gone (specifically, they still remain 30% of the time). additionally, if we decide to crank up the speed further, say to a 12 second target block time, single-level is just not good enough. here's a result with those statistics: ### printing results ### 1 1.0 10 10.4567533177 15 16.3077390517 5 5.0859101624 25 29.6409432377 total blocks produced: 83315 total blocks in chain: 66866 efficiency: 0.802568565084 average uncles: 0.491246459555 length of chain: 44839 block time: 22.3020138719 18% centralization gain. thus, we need a new strategy. a new strategy the first idea i tried about one week ago was requiring every block to have five uncles; this would in a sense decentralize the production of each block further, ensuring that no miner had a clear advantage in making the next block. since the math for that is pretty hopelessly intractable (well, if you try hard at it for months maybe you could come up with something involving nested poisson processes and combinatorical generating functions, but i'd rather not), here's the sim script. note that there are actually two ways you can do the algorithm: require the parent to be the lowest-hash child of the grandparent, or require the parent to be the highest-score child of the grandparent. the first way (to do this yourself, modify line 56 to if newblock["id"] > self.blocks[self.head]["id"]:, we get this: ### printing results ### 1 1.0 10 9.59485744106 25 24.366668248 5 4.82484937616 15 14.0160823568 total blocks produced: 8033 total blocks in chain: 2312 efficiency: 0.287812772314 average uncles: 385.333333333 length of chain: 6 block time: 13333.3333333 ooooops! well, let's try the highest-score model: ### printing results ### 1 1.0 10 9.76531271652 15 14.1038046954 5 5.00654546181 25 23.9234131003 total blocks produced: 7989 total blocks in chain: 6543 efficiency: 0.819001126549 average uncles: 9.06232686981 length of chain: 722 block time: 110.8033241 so here we have a very counterintuitive result: the 25% hashpower mining pool gets only 24x as much as a 1% hashpower pool. economic sublinearity is a cryptoeconomic holy grail, but unfortunately it is also somewhat of a perpetual motion machine; unless you rely on some specific thing that people have a certain amount of (eg. home heating demand, unused cpu power), there is no way to get around the fact even if you come up with some clever sublinear concoction an entity with 25x as much power going in will at the very least be able to pretend to be 25 separate entities and thus claim a 1x reward. thus, we have an unambiguous (okay, fine, 99 point something percent confidence) empirical proof that the 25x miners are acting suboptimally, meaning that the optimal strategy in this environment is not to always mine the block with the highest score. the reasoning here is this: if you mine on a block that has the highest score, then there is some chance that someone else will discover a new uncle one level back, and then mine a block on top of that, creating a new block at the same level as your block but with a slightly higher score and leaving you in the dust. however, if you try to be one of those uncles, then the highest-score block at the next level will certainly want to include you, so you will get the uncle reward. the presence of one non-standard strategy strongly suggests the existence of other, and more exploitative, non-standard strategies, so we're not going this route. however, i chose to include it in the blog post to show an example of what the dangers are. so what is the best way forward? as it turns out, it's pretty simple. go back to single level ghost, but allow uncles to come from up to 5 blocks back. hence, the child of a parent of a parent (hereinafter, -2,+1-ancestor) is a valid uncle, a -3,+1-ancestor is a valid uncle, as is a -4,+1-ancestor and a -5,+1-ancestor, but a -6,+1-ancestor or a -4,+2-ancestor (ie. c(c(p(p(p(p(head)))))) where no simplification is possible) is not. additionally, we increase the uncle reward to 15/16, and cut the nephew reward to 1/32. first, let's make sure that it works under standard strategies. in the ghost sim script, set uncle_depth to 4, pow_solution_time to 12, transit_time to 12, uncle_reward_coeff to 15/16 and nephew_reward_coeff to 1/32 and see what happens: ### printing results ### 1 1.0 10 10.1329810896 25 25.6107014231 5 4.96386947539 15 15.0251826297 total blocks produced: 83426 total blocks in chain: 77306 efficiency: 0.926641574569 average uncles: 0.693116362601 length of chain: 45659 block time: 21.901487111 completely reasonable all around, although note that the actual block time is 21s due to inefficiency and uncles rather than the 12s we targeted. now, let's try a few more trials for enlightenment and fun: uncle_reward_coeff = 0.998, nephew_reward_coeff = 0.001 lead to the 25% mining pool getting a roughly 25.3x return, and setting uncle_reward_coeff = 7/8 and nephew_reward_coeff = 1/16 leads to the 25% mining pool getting a 26.26% return. obviously setting the uncle_reward_coeff all the way to zero would negate the benefit completely, so it's good to have it be as close to one as possible, but if it's too close to one than there's no incentive to include uncles. uncle_reward_coeff = 15/16 seems to be a fair middle ground, giving the 25% miner a 2.5% centralization advantage allowing uncles going back 50 blocks, surprisingly, has fairly little substantial efficiency gain. the reason is that the dominant weakness of -5,+1 ghost is the +1, not the -5, ie. stale c(c(p(p(..p(head)..)))) blocks are the problem. as far as centralization goes, with 0.998/0.001 rewards it knocks the 25% mining pool's reward down to essentially 25.0x. with 15/16 and 1/32 rewards there is no substantial gain over the -4,+1 approach. allowing -4,+3 children increases efficiency to effectively 100%, and cuts centralization to near-zero assuming 0.998/0.001 rewards and has negligible benefit assuming 15/16 and 1/32 rewards. if we reduce the target block time to 3 seconds, efficiency goes down to 66% and the 25% miner gets a 31.5x return (ie. 26% centralization gain). if we couple this with a -50,+1 rule, the effect is negligible (25% -> 31.3x), but if we use a -4,+3 rule efficiency goes up to 83% and the 25% miner only gets a 27.5x return (the way to add this to the sim script is to add after line 65 for c2 in self.children.get(c, {}): u[c2] = true for a -n,+2 rule and then similarly nest down one level further for -n,+3). additionally, the actual block time in all three of these scenarios is around 10 seconds. if we reduce the target block time to 6 seconds, then we get an actual block time of 15 seconds and the efficiency is 82% and the 25% miner gets 26.8x even without improvements. now, let's look at the other two risks of limited ghost that we discussed above: the non-head dominant strategy and the selfish-mining attack. note that there are actually two non-head strategies: try to take more uncles, and try to be an uncle. trying to take more uncles was useful in the -2,+1 case, and trying to be an uncle was useful in the cas of my abortive mandatory-5-uncles idea. trying to be an uncle is not really useful when multiple uncles are not required, since the reason why that alternative strategy worked in the mandatory-5-uncle case is that a new block is useless for further mining without siblings. thus, the only potentially problematic strategy is trying to include uncles. in the one-block case, it was a problem, but here is it not because most uncles that can be included after n blocks can also be included after n+1 blocks, so the practical extent to which it will matter is limited. the selfish-mining attack also no longer works for a similar reason. if you fail to include uncles, then the guy after you will. there are four chances for an uncle to get in, so not including uncles is a 4-party prisoner's dilemma between anonymous players a game that is doomed to end badly for everyone involved (except of course the uncles themselves). there is also one last concern with this strategy: we saw that rewarding all uncles makes 51% attacks cost-free, so are they cost-free here? beyond one block, the answer is no; although the first block of an attempted fork will get in as an uncle and receive its 15/16x reward, the second and third and all subsequent ones will not, so starting from two confirmations attacks still cost miners almost as much as they did before. twelve seconds, really? the most surprising finding about decker and wattenhofer's finding is the sheer length of time that blocks take to propagate an amazingly slow 12 seconds. in decker and wattenhofer's analysis, the 12 second delay is actually mostly because of the need to download and verify the blocks themselves; ie. the algorithm that bitcoin clients follow is: def on_receive_block(b): if not verify_pow_and_header(b): return if not verify_transactions(b): return accept(b) start_broadcasting(b) however, decker and wattenhofer did propose a superior strategy which looks something like this: def on_receive_header(h): if not verify_pow_and_header(h): return ask_for_full_block(h, callback) start_broadcasting(h) def callback(b): start_broadcasting(b) if not verify_transactions(b): stop_broadcasting(b) return accept(b) this allows all of the steps to happen in parallel; headers can get broadcasted first, then blocks, and the verifications do not need to all be done in series. although decker and wattenhofer do not provide their own estimate, intuitively this seems like it may speed up propagation by 25-50%. the algorithm is still non-exploitable because in order to produce an invalid block that passes the first check a miner would still need to produce a valid proof of work, so there is nothing that the miner could gain. another point that the paper makes is that the transit time is, beyond a certain point, proportional to block size; hence, cutting block size by 50% will also cut transit time to something like 25-40%; the nonscaling portion of the transit time is something like 2s. hence, a 3-second target block time (and 5s actual block time) may be quite viable. as usual, we'll be more conservative at first and not take things that far, but a block time of 12s does nevertheless seem to be very much achievable. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements ethereum community and adoption update week 1 | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search ethereum community and adoption update week 1 posted by stephan tual on november 3, 2014 research & development hi, i'm stephan tual, and i've been responsible for ethereum's adoption and education since january as cco. i'm also leading our uk ðξv hub, located at co-work in putney (south west london). i feel really privileged to be able to lead the effort on the communication strategy at ðξv. for the very first time, we’re seeing the mainstream public take a genuine interest in the potential of decentralisation. the feeling of excitement about what ‘could be’ when i first read vitalik’s whitepaper on that fateful christmas afternoon is now shared by dozens of thousands of technologists, developers and entrepreneurs. thanks to the ether sale, a group of smart, hardworking individuals is now able to work full time on solving core technical and adoption challenges, and to deliver a solution at 10x the speed an equivalent garage-based initiative would have taken. with ethereums’ apis supporting gav’s vision for web 3, it finally is in the reach of the community to build decentralized applications without middlemen. by democratizing access to programmable blockchain technology, ethereum empowers software developers and entrepreneurs to make a major impact on not only the decentralisation of the economy, but also social structures, voting mechanisms and so much more. it’s a very ambitious project, and everyone at ðξv feels a strong sense of duty to to deliver on this vision. as part of our efforts, technology is of course key, but so is adoption. ethereum without dapps (decentralized apps) would be akin to a video game console without launch titles, and, just like any protocol, we expect the applications to be the real stars of the show. here’s how we plan to spread the word and support developers in their efforts. education building a curriculum: we’re building an extensive curriculum adapted to both teachers and self-learners at home, at hackathons and in universities around the globe. consisting of well defined modules progressing over time in their complexity, our goal is to establish a learning standard that will be of course completely free of charge and 100% open source. content aggregation: at the moment we are aware that information on how to ‘get into’ ethereum is a little bit fragmented between forums, multiple wikis and various 3rd party sites. a subdomain to our website will be created during the course of the next few months to access this valuable information easily and in one place. produce tutorials, videos and articles: tutorials are key to learn a new set of languages and tools. by producing both videos and text-based tutorials, we intend to give the community an insiders’s view on best practices, from structuring contract storage to leveraging the new whisper p2p messaging system for example. codeacademy-like site: not everyone likes to learn within a classroom environment, and some feel constrained by linear tutorials. with a release date coinciding with the launch of ethereum, we’re partnering with a us-based company to build a codeacademy-like site within a gamified environment, where you'll be able to learn at your own pace how to build dapps and their backend contracts. university chapters: vitalik and i recently gave a presentation at cambridge university and ethereum will participate in the hackathon on transparency on november 26 at the university of geneva. encouraged by the enthusiasm we’ve witnessed in the academic world, we are working to support directly the oxbridge blocktech network (obn) in their efforts to build a network of chapters, firstly within the uk then throughout europe. i’m happy to announce that ken kappler has joined the uk team to help with these educational efforts. many of you in london know ken as he’s been a semi-permanent fixture at all our meetups and hackathons, kindly helping behind the scenes. ken, known as bluechain on irc, is also the writer behind http://dappsforbeginners.wordpress.com/ which will soon merge with our own education site. ken will lead a weekly ‘ethereum clinic’ on irc to answer any questions you might have with your current project. times will be posted on our forums. meetups encouraging the creation of new meetups: we now have an extensive network of 85 meetups worldwide, which is an amazing achievement but not sufficient to handle the overwhelming demand for regular catchups in a format that’s appropriate for the local needs and culture. we intend to encourage the creation of new meetups in almost every country and major urban hubs. tooling and support: in order to drive the effort to create and maintain such a large network of international meetups, we will be providing tools for meetup leaders to interact with each other, gain access to the core dev team for video-conference or physical interventions, and exchange information about speakers. these tools will of course be free to use and access. collaterals and venues: for the meetups that are the most active, ethereum is considering, where appropriate, the use of small bursaries so that meetup leaders in these ‘core locales’ do not have to contend with the full costs of collaterals and venues. we will also work with our partners to help meetups secure sponsorships and access to free locations to hold their mini-conferences. global hackathon: starting this week, the ethereum workshops are going to slowly transform into proper hackathons. we are working with wonderful locations around the globe, the vast majority of which started off as ethereum meetups, to organize a worldwide hackathon with some great eth prizes for the best dapps. we're very lucky to welcome anthony d’onofrio to drive these very important initiatives. anthony starts on the 10th of this month and will also cover the north american region from a community perspective you probably already know him as 'texture’, his handle on most forums and channels. community i’m incredibly proud that ethereum’s exposure in the community has been entirely organic since day one. this has been the result of major, time consuming efforts to identify ethereum projects in the wild, reaching out directly and building a strong relationship with our user base. we have achieved several key milestones, including over 10,000 followers on twitter, 100,000 page views per month on our website, similar numbers on our forums and the growth trend only continues to accelerate. historically, ethereum has never used pr as a tool to increase adoption, relying instead on word of mouth, meetups and conferences to spread the word. as the media attention is now intensifying rapidly, i’m pleased to welcome freya stevens to the team as pr/marketing lead. freya will help us build a shared database of media leads, write articles and make complex technology palatable to the general public while identifying strong story angles. freya is based out of cambridge, uk. also with a view to to scale up these initiatives we’re proud to welcome george hallam to the team, aka thehighfiveghost, who recently posted a survey so you can let ethereum know how well we’re doing our job as custodians and developers of the platform. george, as a key supporter to the london community, will be already be familiar to many. as part of these efforts, expect to see a lot more interactions on reddit, irc, discuss and of course our very own forums. george will also help me identify key ethereum-based projects and make contact to see how we can best help with information, connections and inclusion as guests in our weekly video updates, shot at our putney studio. in order to produce very high quality content, we are also welcoming ian meikle to the london hub. ian is the creator of most of the video materials you might have seen relating to ethereum, including the superb video loop that has been a staple at many ethereum meetups. ian will leverage the equipment at our studio to create explainer videos, interview key players in the space, and record panels led by vinay gupta, who joins the coms team as strategic consultant. in london, and above and beyond our existing panels, socials and hackathons, regular 'show and tell' are being scheduled for dapp developers to present their work and receive feedback, a model we intend to promote internationally within the month. and of course, last but not least, expect a major refresh to our website, with beautiful, clean content, practical examples of dapps, a dynamic meetup map and links to all our newly created assets and community points of contacts. in conclusion the question we’re going to continue asking ourselves everyday is how do we support you, the community, in building kick-ass dapps and being successful in your venture on our platform. i hope the above gives you a quick intro as to our plans. i’ll be issuing regular updates both on this blog and on our youtube. stephan (stephan.nospam.tual@ethdev.com) previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle engineering security through coordination problems 2017 may 08 see all posts recently, there was a small spat between the core and unlimited factions of the bitcoin community, a spat which represents perhaps the fiftieth time the same theme was debated, but which is nonetheless interesting because of how it highlights a very subtle philosophical point about how blockchains work. viabtc, a mining pool that favors unlimited, tweeted "hashpower is law", a usual talking point for the unlimited side, which believes that miners have, and should have, a very large role in the governance of bitcoin, the usual argument for this being that miners are the one category of users that has a large and illiquid financial incentive in bitcoin's success. greg maxwell (from the core side) replied that "bitcoin's security works precisely because hash power is not law". the core argument is that miners only have a limited role in the bitcoin system, to secure the ordering of transactions, and they should not have the power to determine anything else, including block size limits and other block validity rules. these constraints are enforced by full nodes run by users if miners start producing blocks according to a set of rules different than the rules that users' nodes enforce, then the users' nodes will simply reject the blocks, regardless of whether 10% or 60% or 99% of the hashpower is behind them. to this, unlimited often replies with something like "if 90% of the hashpower is behind a new chain that increases the block limit, and the old chain with 10% hashpower is now ten times slower for five months until difficulty readjusts, would you really not update your client to accept the new chain?" many people often argue against the use of public blockchains for applications that involve real-world assets or anything with counterparty risk. the critiques are either total, saying that there is no point in implementing such use cases on public blockchains, or partial, saying that while there may be advantages to storing the data on a public chain, the business logic should be executed off chain. the argument usually used is that in such applications, points of trust exist already there is someone who owns the physical assets that back the on-chain permissioned assets, and that someone could always choose to run away with the assets or be compelled to freeze them by a government or bank, and so managing the digital representations of these assets on a blockchain is like paying for a reinforced steel door for one's house when the window is open. instead, such systems should use private chains, or even traditional server-based solutions, perhaps adding bits and pieces of cryptography to improve auditability, and thereby save on the inefficiencies and costs of putting everything on a blockchain. the arguments above are both flawed in their pure forms, and they are flawed in a similar way. while it is theoretically possible for miners to switch 99% of their hashpower to a chain with new rules (to make an example where this is uncontroversially bad, suppose that they are increasing the block reward), and even spawn-camp the old chain to make it permanently useless, and it is also theoretically possible for a centralized manager of an asset-backed currency to cease honoring one digital token, make a new digital token with the same balances as the old token except with one particular account's balance reduced to zero, and start honoring the new token, in practice those things are both quite hard to do. in the first case, users will have to realize that something is wrong with the existing chain, agree that they should go to the new chain that the miners are now mining on, and download the software that accepts the new rules. in the second case, all clients and applications that depend on the original digital token will break, users will need to update their clients to switch to the new digital token, and smart contracts with no capacity to look to the outside world and see that they need to update will break entirely. in the middle of all this, opponents of the switch can create a fear-uncertainty-and-doubt campaign to try to convince people that maybe they shouldn't update their clients after all, or update their client to some third set of rules (eg. changing proof of work), and this makes implementing the switch even more difficult. hence, we can say that in both cases, even though there theoretically are centralized or quasi-centralized parties that could force a transition from state a to state b, where state b is disagreeable to users but preferable to the centralized parties, doing so requires breaking through a hard coordination problem. coordination problems are everywhere in society and are often a bad thing while it would be better for most people if the english language got rid of its highly complex and irregular spelling system and made a phonetic one, or if the united states switched to metric, or if we could immediately drop all prices and wages by ten percent in the event of a recession, in practice this requires everyone to agree on the switch at the same time, and this is often very very hard. with blockchain applications, however, we are doing something different: we are using coordination problems to our advantage, using the friction that coordination problems create as a bulwark against malfeasance by centralized actors. we can build systems that have property x, and we can guarantee that they will preserve property x to a high degree because changing the rules from x to not-x would require a whole bunch of people to agree to update their software at the same time. even if there is an actor that could force the change, doing so would be hard. this is the kind of security that you gain from client-side validation of blockchain consensus rules. note that this kind of security relies on the decentralization of users specifically. even if there is only one miner in the world, there is still a difference between a cryptocurrency mined by that miner and a paypal-like centralized system. in the latter case, the operator can choose to arbitrarily change the rules, freeze people's money, offer bad service, jack up their fees or do a whole host of other things, and the coordination problems are in the operator's favor, as such systems have substantial network effects and so very many users would have to agree at the same time to switch to a better system. in the former case, client-side validation means that many attempts at mischief that the miner might want to engage in are by default rejected, and the coordination problem now works in the users' favor. note that the arguments above do not, by themselves, imply that it is a bad idea for miners to be the principal actors coordinating and deciding the block size (or in ethereum's case, the gas limit). it may well be the case that, in the specific case of the block size/gas limit, "government by coordinated miners with aligned incentives" is the optimal approach for deciding this one particular policy parameter, perhaps because the risk of miners abusing their power is lower than the risk that any specific chosen hard limit will prove wildly inappropriate for market conditions a decade after the limit is set. however, there is nothing unreasonable about saying that government-by-miners is the best way to decide one policy parameter, and at the same saying that for other parameters (eg. block reward) we want to rely on client-side validation to ensure that miners are constrained. this is the essence of engineering decentralized instutitions: it is about strategically using coordination problems to ensure that systems continue to satisfy certain desired properties. the arguments above also do not imply that it is always optimal to try to put everything onto a blockchain even for services that are trust-requiring. there generally are at least some gains to be made by running more business logic on a blockchain, but they are often much smaller than the losses to efficiency or privacy. and this ok; the blockchain is not the best tool for every task. what the arguments above do imply, though, is that if you are building a blockchain-based application that contains many centralized components out of necessity, then you can make substantial further gains in trust-minimization by giving users a way to access your application through a regular blockchain client (eg. in the case of ethereum, this might be mist, parity, metamask or status), instead of getting them to use a web interface that you personally control. theoretically, the benefits of user-side validation are optimized if literally every user runs an independent "ideal full node" a node that accepts all blocks that follow the protocol rules that everyone agreed to when creating the system, and rejects all blocks that do not. in practice, however, this involves asking every user to process every transaction run by everyone in the network, which is clearly untenable, especially keeping in mind the rapid growth of smartphone users in the developing world. there are two ways out here. the first is that we can realize that while it is optimal from the point of view of the above arguments that everyone runs a full node, it is certainly not required. arguably, any major blockchain running at full capacity will have already reached the point where it will not make sense for "the common people" to expend a fifth of their hard drive space to run a full node, and so the remaining users are hobbyists and businesses. as long as there is a fairly large number of them, and they come from diverse backgrounds, the coordination problem of getting these users to collude will still be very hard. second, we can rely on strong light client technology. there are two levels of "light clients" that are generally possible in blockchain systems. the first, weaker, kind of light client simply convinces the user, with some degree of economic assurance, that they are on the chain that is supported by the majority of the network. this can be done much more cheaply than verifying the entire chain, as all clients need to do is in proof of work schemes verify nonces or in proof stake schemes verify signed certificates that state "either the root hash of the state is what i say it is, or you can publish this certificate into the main chain to delete a large amount of my money". once the light client verifies a root hash, they can use merkle trees to verify any specific piece of data that they might want to verify. look, it's a merkle tree! the second level is a "nearly fully verifying" light client. this kind of client doesn't just try to follow the chain that the majority follows; rather, it also tries to follow only chains that follow all the rules. this is done by a combination of strategies; the simplest to explain is that a light client can work together with specialized nodes (credit to gavin wood for coming up with the name "fishermen") whose purpose is to look for blocks that are invalid and generate "fraud proofs", short messages that essentially say "look! this block has a flaw over here!". light clients can then verify that specific part of a block and check if it's actually invalid. if a block is found to be invalid, it is discarded; if a light client does not hear any fraud proofs for a given block for a few minutes, then it assumes that the block is probably legitimate. there's a bit more complexity involved in handling the case where the problem is not data that is bad, but rather data that is missing, but in general it is possible to get quite close to catching all possible ways that miners or validators can violate the rules of the protocol. note that in order for a light client to be able to efficiently validate a set of application rules, those rules must be executed inside of consensus that is, they must be either part of the protocol or part of a mechanism executing inside the protocol (like a smart contract). this is a key argument in favor of using the blockchain for both data storage and business logic execution, as opposed to just data storage. these light client techniques are imperfect, in that they do rely on assumptions about network connectivity and the number of other light clients and fishermen that are in the network. but it is actually not crucial for them to work 100% of the time for 100% of validators. rather, all that we want is to create a situation where any attempt by a hostile cartel of miners/validators to push invalid blocks without user consent will cause a large amount of headaches for lots of people and ultimately require everyone to update their software if they want to continue to synchronize with the invalid chain. as long as this is satisfied, we have achieved the goal of security through coordination frictions. on stake | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on stake posted by vitalik buterin on july 5, 2014 research & development the topic of mining centralization has been a very important one over the past few weeks. ghash.io, the bitcoin network's largest mining pool, has for the past month directed over 40% of the bitcoin network's hashpower, and two weeks ago briefly spiked over 50%, theoretically giving it monopoly control over the bitcoin network. although miners quidkly left the pool and reduced its hashpower to 35%, it's clear that the problem is not solved. at the same time, asics threaten to further centralize the very production . one approach to solving the problem is the one i advocated in my previous post: create a mining algorithm that is guaranteed to remain cpu-friendly in the long term. another, however, is to abolish mining entirely, and replace it with a new model for seeking consensus. the primary second contender to date has been a strategy called "proof of stake", the intuition behind which is as follows. in a traditional proof-of-work blockchain, miners "vote" on which transactions came at what time with their cpu power, and the more cpu power you have the proportionately larger your influence is. in proof-of-stake, the system follows a similar but different paradigm: stakeholders vote with their dollars (or rather, the internal currency of the particular system). in terms of how this works technically, the simplest setup is a model that has been called the "simulated mining rig": essentially, every account has a certain chance per second of generating a valid block, much like a piece of mining hardware, and this chance is proportional to the account's balance. the simplest formula for this is: sha256(prevhash + address + timestamp) <= 2^256 * balance / diff prevhash is the hash of the previous block, address is the address of the stake-miner, timestamp is the current unix time in second, balance is the account balance of the stack-miner and diff is an adjustable global difficulty parameter. if a given account satisfies this equation at any particular second, it may produce a valid block, giving that account some block reward. another approach is to use not the balance, but the "coin age" (ie. the balance multiplied by the amount of time that the coins have not been touched), as the weighting factor; this guarantees more even returns but at the expense of potentially much easier collusion attacks, since attackers have the ability to accumulate coin age, and possible superlinearity; for these reasons, i prefer the plain balance-based approach in most cases, and we will use this as our baseline for the rest of this discussion. other solutions to "proof of x" have been proposed, including excellence, bandwidth, storage and identity, but none are particularly convenient as consensus algorithms; rather, all of these systems have many of the same properties of proof of stake, and are thus best implemented indirectly by making them purely mechanisms for currency distribution, and then using proof of stake on those distributed coins for the actual consensus. the only exception is perhaps the social-graph-theory based ripple, although many cryptocurrency proponents consider such systems to be far too trust-dependent in order to be considered truly "decentralized"; this point can be debated, but it is best to focus on one topic at a time and so we will focus on stake. strengths and weaknesses if it can be implemented correctly, in theory proof of stake has many advantages. in particular are three: it does not waste any significant amount of electicity. sure, there is a need for stakeholders to keep trying to produce blocks, but no one gains any benefit from making more than one attempt per account per second; hence, the electricity expenditure is comparable to any other non-wasteful internet protocol (eg. bittorrent) it can arguably provide a much higher level of security. in proof of work, assuming a liquid market for computing power the cost of launching a 51% attack is equal to the cost of the computing power of the network over the course of two hours an amount that, by standard economic principles, is roughly equal to the total sum of block rewards and transaction fees provided in two hours. in proof of stake, the threshold is theoretically much higher: 51% of the entire supply of the currency. depending on the precise algorithm in question it can potentially allow for much faster blockchains (eg. nxt has one block every few seconds, compared to one per minute for ethereum and one per ten minutes for bitcoin) note that there is one important counterargument that has been made to #2: if a large entity credibly commits to purchasing 51% of currency units and then using those funds to repeatedly sabotage the network, then the price will fall drastically, making it much easier for that entity to puchase the tokens. this does somewhat mitigate the benefit of stake, although not nearly fatally; an entity that can credibly commit to purchasing 50% of coins is likely also one that can launch 51% attacks against proof of work. however, with the naive proof of stake algorithm described above, there is one serious problem: as some bitcoin developers describe it, "there is nothing at stake". what that means is this: in the context of a proof-of-work blockchain, if there is an accidental fork, or a deliberate transaction reversal ("double-spend") attempt, and there are two competing forks of the blockchain, then miners have to choose which one they contribute to. their three choices are either: mine on no chain and get no rewards mine on chain a and get the reward if chain a wins mine on chain b and get the reward if chain b wins as i have commented in a previous post, note the striking similarity to schellingcoin/truthcoin here: you win if you go with what everyone else goes with, except in this case the vote is on the order of transactions, not a numerical (as in schellingcoin) or binary (as in truthcoin) datum. the incentive is to support the chain that everyone else supports, forcing rapid convergence, and preventing successful attacks provided that at least 51% of the network is not colluding. in the naive proof of stake algorithm, on the other hand, the choices of whether or not to vote on a and whether or not to vote on b are independent; hence, the optimal strategy is to mine on any fork that you can find. thus, in order to launch a successful attack, an attacker need only overpower all of the altruists who are willing to vote only on the correct chain. the problem is, unfortunately, somewhat fundamental. proof of work is nice because the property of hash verification allows the network to be aware of something outside of itself namely, computing power, and that thing serves as a sort of anchor to ensure some stability. in a naive proof of stake system, however, the only thing that each chain is aware of is itself; hence, one can intuitively see that this makes such systems more flimsy and less stable. however, the above is merely an intuitive argument; it is by no means a mathematical proof that a proof-of-stake system cannot be incentive-compatible and secure, and indeed there are a number of potential ways to get around the issue. the first strategy is the one that is employed in the slasher algorithm, and it hinges on a simple realization: although, in the case of a fork, chains are not aware of anything in the outside world, they are aware of each other. hence, the way the protocol prevents double-mining is this: if you mine a block, the reward is locked up for 1000 blocks, and if you also mine on any other chain then anyone else can submit the block from the other chain into the original chain in order to steal the mining reward. note, however, that things are not quite so simple, and there is one catch: the miners have to be known in advance. the problem is that if the algorithm given above is used directly, then the issue arises that, using a probabilistic strategy, double mining becomes very easy to hide. the issue is this: suppose that you have 1% stake, and thus every block there is a 1% chance that you will be able to produce (hereinafter, "sign") it. now, suppose there is a fork between chain a and chain b, with chain a being the "correct" chain. the "honest" strategy is to try to generate blocks just on a, getting an expected 0.01 a-coins per block. an alternative strategy, however, is to try to generate blocks on both a and b, and if you find a block on both at the same time then discarding b. the payout per block is one a-coin if you get lucky on a (0.99% chance), one b-coin if you get lucky on b (0.99% chance) and one a-coin, but no b-coins, if you get lucky on both; hence, the expected payout is 0.01 a-coins plus 0.0099 b-coins if you double-vote. if the stakeholders that need to sign a particular block are decided in advance, however (ie. specifically, decided before a fork starts), then there is no possibility of having the opportunity to vote on a but not b; you either have the opportunity on both or neither. hence, the "dishonest" strategy simply collapses into being the same thing as the "honest" strategy. the block signer selection problem but then if block signers are decided in advance, another issue arises: if done wrong, block signers could "mine" their blocks, repeatedly trying to create a block with different random data until the resulting block triggers that same signer having the privilege to sign a block again very soon. for example, if the signer for block n+1000 was simply chosen from the hash of block n, and an attacker had 1% stake, then the attacker could keep rebuilding the block until block n+1000 also had the attacker as its signer (ie. an expected 100 iterations). over time, the attacker would naturally gain signing privilege on other blocks, and thus eventually come to completely saturate the blockchain with length-1000 cycles controlled by himself. even if the hash of 100 blocks put together is used, it's possible to manipulate the value. thus, the question is, how do we determine what the signers for future blocks are going to be? the solution used in slasher is to use a secure decentralized random number generator protocol: many parties come in, first submit to the blockchain the hashes of their values, and then submit their values. there is no chance of manipulation this way, because each submitter is bound to submit in the second round the value whose hash they provided in the first round, and in the first round no one has enough information in order to engage in any manipulation. the player still has a choice of whether or not to participate in the second round, but the two countervailing points are that (1) this is only one bit of freedom, although it becomes greater for large miners that can control multiple accounts, and (2) we can institute a rule that failing to participate causes forfeiture of one's mining privilege (miners in round n choose miners for round n+1 during round n-1, so there is an opportunity to do this if certain round-n miners misbehave during this selection step). another idea, proposed by iddo bentov and others in their "cryptocurrencies without proof of work" paper, is to use something called a "low-influence" function essentially, a function such that there is only a very low chance that a single actor will be able to change the result by changing the input. a simple example of an lif over small sets is majority rule; here, because we are trying to pick a random miner, we have a very large set of options to choose from, so majority rule per bit is used (eg. if you have 500 parties and you want to pick a random miner out of a billion, assign them into thirty groups of 17, and have each group vote on whether their particular bit is zero or one, and then recombine the bits as a binary number at the end). this removes the need for a complicated two-step protocol, allowing it to potentially be done much more quickly and even in parallel, reducing the risk that the pre-chosen stake-miners for some particular block will get together and collude. a third interesting strategy, used by nxt, is to use the addresses of the stake-miners for blocks n and n+1 to choose the miner for block n+2; this by definition gives only one choice for the next miner in each block. adding a criterion that every miner needs to be locked in for 1440 blocks in order to participate prevents sending transactions as a form of double-mining. however, having such rapid stake-miner selection also compromises the nothing-at-stake resistance property due to the probabilistic double-mining problem; this is the reason why clever schemes to make miner determination happen very quickly are ultimately, beyond a certain point, undesirable. long-range attacks while the slasher approach does effectively solve the nothing-at-stake problem against traditional 51% attacks, a problem arises in the case of something called a "long-range attack": instead of an attacker starting mining from ten blocks before the current block, the attacker starts ten thousand blocks ago. in a proof-of-work context, this is silly; it basically means doing thousands of times as much work as is necessary to launch an attack. here, however, creating a block is nearly computationally free, so it's a reasonable strategy. the reason why it works is that slasher's process for punishing multi-mining only lasts for 1000 blocks, and its process for determining new miners lasts 3000 blocks, so outside the "scope" of that range slasher functions exactly like the naive proof-of-stake coin. note that slasher is still a substantial improvement; in fact, assuming users never change it can be made fully secure by introducing a rule into each client not to accept forks going back more than 1000 blocks. the problem is, however, what happens when a new user enters the picture. when a new user downloads a proof-of-stake-coin client for the first time, it will see multiple versions of the blockchain: the longest, and therefore legitimate, fork, and many pretenders trying to mine their own chains from the genesis. as described above, proof-of-stake chains are completely self-referential; hence, the client seeing all of these chains has no idea about any surrounding context like which chain came first or which has more value (note: in a hybrid proof-of-stake plus social graph system, the user would get initial blockchain data from a trusted source; this approach is reasonable, but is not fully decentralized). the only thing that the client can see is the allocation in the genesis block, and all of the transactions since that point. thus, all "pure" proof-of-stake systems are ultimately permanent nobilities where the members of the genesis block allocation always have the ultimate say. no matter what happens ten million blocks down the road, the genesis block members can always come together and launch an alternate fork with an alternate transaction history and have that fork take over. if you understand this, and you are still okay with pure proof of stake as a concept (the specific reason why you might still be okay is that, if the initial issuance is done right, the "nobility" should still be large enough that it cannot practically collude), then the realization allows for some more imaginative directions in terms of how proof of stake can play out. the simplest idea is to have the members of the genesis block vote on every block, where double-mining is punished by permanent loss of voting power. note that this system actually solves nothing-at-stake issues completely, since every genesis block holder has a mining privilege that has value forever into the future, so it will always not be worth it to double-mine. this system, however, has a finite lifespan specifically, the maximum life (and interest) span of the genesis signers, and it also gives the nobility a permanent profit-making privilege, and not just voting rights; however, nevertheless the existence of the algorithm is encouraging because it suggests that long-range-nothing-at-stake might be fundamentally resolvable. thus, the challenge is to figure out some way to make sure voting privileges transfer over, while still at the same time maintaining security. changing incentives another approach to solving nothing-at-stake comes at the problem from a completely different angle. the core problem is, in naive proof-of-stake, rational individuals will double-vote. the slasher-like solutions all try to solve the problem by making it impossible to double-vote, or at the very least heavily punishing such a strategy. but what if there is another approach; specifically, what if we instead remove the incentive to do so? in all of the proof of stake systems that i described above, the incentive is obvious, and unfortunately fundamental: because whoever is producing blocks needs an incentive to participate in the process, they benefit if they include a block in as many forks as possible. the solution to this conundrum comes from an imaginative, out-of-the-box proposal from daniel larimer: transactions as proof of stake. the core idea behind transactions as proof-of-stake is simple: instead of mining being done by a separate class of individuals, whether computer hardware owners or stakeholders, mining and transaction sending are merged into one. the naive tapos algorithm is as follows: every transaction must contain a reference (ie. hash) to the previous transaction a candidate state-of-the-system is obtained by calculating the result of a resulting transaction chain the correct chain among multiple candidates is the one that has either (i) the longest coin-days-destroyed (ie. number of coins in the account * time since last access), or (ii) the highest transaction fees (these are two different options that we will analyze separately) this algorithm has the property that it is extremely unscalable, breaking down beyond about 1 transaction per 2-5 seconds, and it is not the one that larimer suggests or the one that will actually be used; rather, it's simply a proof of concept that we will analyze to see if this approach is valid at all. if it is, then there are likely ways to optimize it. now, let's see what the economics of this are. suppose that there is a fork, and there are two competing versions of the tapos chain. you, as a transaction sender, made a transaction on chain a, and there is now an upcoming chain b. do you have the incentive to double-mine and include your transaction in chain b as well? the answer is no in fact you actually want to double-spend your recipient so you would not put the transaction on another chain. this argument is especially potent in the case of long-range attacks, where you already received your product in exchange for the funds; in the short term, of course, the incentive still exists to make sure the transaction is sent, so senders do have the incentive to double-mine; however, because the worry is strictly time-limited this can be resolved via a slasher-like mechanism. one concern is this: given the presence of forks, how easy is it to overwhelm the system? if, for example, there is a fork, and one particular entity wants to double-spend, under what circumstances is that possible? in the transaction-fee version, the requirement is pretty simple: you need to spend more txfees than the rest of the network. this seems weak, but in reality it isn't; we know that in the case of bitcoin, once the currency supply stops increasing mining will rely solely on transaction fees, and the mechanics are exactly the same (since the amount that the network will spend on mining will roughly correspond to the total number of txfees being sent in); hence, fee-based tapos is in this regard at least as secure as fee-only pow mining. in the second case, we have a different model: instead of mining with your coins, you are mining with your liquidity. anyone can 51% attack the system if and only if they have a sufficiently large quantity of coin-days-destroyed on them. hence, the cost of spending a large txfee after the fact is replaced by the cost of sacrificing liquidity before the fact. cost of liquidity the discussion around liquidity leads to another important philosophical point: security cannot be cost-free. in any system where there is a block reward, the thing that is the prerequisite for the reward (whether cpu, stake, or something else) cannot be free, since otherwise everyone would be claiming the reward at infinitum, and in tapos transaction senders need to be providing some kind of fee to justify security. furthermore, whatever resource is used to back the security, whether cpu, currency sacrifices or liquidity sacrifices, the attacker need only get their hands on the same quantity of that resource than the rest of the network. note that, in the case of liquidity sacrifices (which is what naive proof of stake is), the relevant quantity here is actually not 50% of coins, but rather the privilege of accessing 50% of coins for a few hours a service that, assuming a perfectly efficient market, might only cost a few hundred thousand dollars. the solution to this puzzle is that marginal cost is not the same thing as average cost. in the case of proof of work, this is true only to a very limited extent; although miners do earn a positive nonzero profit from mining, they all pay a high cost (unless they're cpu miners heating their homes, but even there there are substantial efficiency losses; laptops running hash functions at 100%, though effective at heating, are necessarily less effective than systems designed for the task). in the case of currency sacrifices, everyone pays the same, but the payment is redistributed as a dividend to everyone else, and this profit is too dispersed to be recovered via market mechanisms; thus, although the system is costly from a local perspective, it is costless from a global perspective. the last option, liquidity sacrifice, is in between the two. although liquidity sacrifice is costly, there is a substantial amount of disparity in how much people value liquidity. some people, like individual users or businesses with low savings, heavily value liquidity; others, like savers, do not value liquidity at all (eg. i could not care less if i lost the ability to spend ten of my bitcoins for some duration). hence, although the marginal cost of liquidity will be high (specifically, necessarily equal to either the mining reward or the transaction fee), the average cost is much lower. hence, there is a leverage effect that allows the cost of an attack to be much higher than the inefficiency of the network, or the amount that senders spend on txfees. additionally, note that in larimer's scheme specifically, things are rigged in such a a way that all liquidity that is sacrificed in consensus is liquidity that was being sacrificed anyway (namely, by not sending coins earlier), so the practical level of inefficiency is zero. now, tapos does have its problems. first, if we try to make it more scalable by reintroducing the concept of blocks, then there ideally needs to be some reason to produce blocks that is not profit, so as not to reintroduce the nothing-at-stake problem. one approach may be to force a certain class of large transaction senders to create blocks. second, attacking a chain is still theoretically "cost-free", so the security assurances are somewhat less nice than they are in proof-of-work. third, in the context of a more complicated blockchain like ethereum, and not a currency, some transactions (eg. finalizing a bet) are actually profitable to send, so there will be incentive to double-mine on at least some transactions (though not nearly all, so there is still some security). finally, it's a genesis-block-nobility system, just like all proof-of-stake necessarily is. however, as far as pure proof-of-stake systems go, it does seem a much better backbone than the version of proof of stake that emulated bitcoin mining. hybrid proof of stake given the attractiveness of proof of stake as a solution for increasing efficiency and security, and its simultaneous deficiencies in terms of zero-cost attacks, one moderate solution that has been brought up many times is hybrid proof of stake, in its latest incantation called "proof of activity". the idea behind proof of activity is simple: blocks are produced via proof of work, but every block randomly assigns three stakeholders that need to sign it. the next block can only be valid once those signatures are in place. in this system, in theory, an attacker with 10% stake would see 999 of his 1000 blocks not being signed, whereas in the legitimate network 729 out of 1000 blocks would be signed; hence, such an attacker would be penalized in mining by a factor of 729. however, there is a problem: what motivates signers to sign blocks on only one chain? if the arguments against pure proof of stake are correct, then most rational stake-miners would sign both chains. hence, in hybrid pos, if the attacker signs only his chain, and altruists only sign the legitimate chain, and everyone else signs both, then if the attacker can overpower the altruists on the stake front that means that the attacker can overtake the chain with less than a 51% attack on the mining front. if we trust that altruists as a group are more powerful in stake than any attacker, but we don't trust that too much, then hybrid pos seems like a reasonable hedge option; however, given the reasoning above, if we want to hybridize one might ask if hybrid pow + tapos might not be the more optimal way to go. for example, one could imagine a system where transactions need to reference recent blocks, and a blockchain's score is calculated based on proof of work and coin-days-destroyed counts. conclusion will we see proof of stake emerge as a viable alternative to proof of work in the next few years? it may well be. from a pure efficiency perspective, if bitcoin, or ethereum, or any other pow-based platform get to the point where they have similar market cap to gold, silver, the usd, eur or cny, or any other mainstream asset, then over a hundred billion dollars worth of new currency units will be produced per year. under a pure-pow regime, an amount of economic power approaching that will be spent on hashing every year. thus, the cost to society of maintaining a proof-of-work cryptocurrency is about the same as the cost of maintaining the russian military (the analogy is particularly potent because militaries are also proof of work; their only value to anyone is protecting against other militaries). under hybrid-pos, that might safely be dropped to $30 billion per year, and under pure pos it would be almost nothing, except depending on implementation maybe a few billion dollars of cost from lost liquidity. ultimately, this boils down to a philosophical question: exactly how much does decentralization mean to us, and how much are we willing to pay for it? remember that centralized databases, and even quasi-centralized ones based on ripple consensus, are free. if perfect decentralization is indeed worth 100billion,thenproofofworkisdefinitelytherightwaytogo.butarguablythatisnotthecase.whatifsocietydoesnotseedecentralizationasagoalinitself,andtheonlyreasonwhyit′sworthittodecentralizeistogettheincreasedbenefitsofefficiencythatdecentralizationbrings?inthatcase,ifdecentralizationcomeswitha100 billion, then proof of work is definitely the right way to go. but arguably that is not the case. what if society does not see decentralization as a goal in itself, and the only reason why it's worth it to decentralize is to get the increased benefits of efficiency that decentralization brings? in that case, if decentralization comes with a 100billion,thenproofofworkisdefinitelytherightwaytogo.butarguablythatisnotthecase.whatifsocietydoesnotseedecentralizationasagoalinitself,andtheonlyreasonwhyit′sworthittodecentralizeistogettheincreasedbenefitsofefficiencythatdecentralizationbrings?inthatcase,ifdecentralizationcomeswitha100 billion price tag, then we should just centralize and let a few governments run the databases. but if we have a solid, viable proof of stake algorithm, then we have a third option: a system which is both decentralized and cost-free (note that useful proof of work also fits this criterion, and may be easier); in that case, the dichotomy does not exist at all and decentralization becomes the obvious choice. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle [mirror] a proof of stake design philosophy 2016 dec 29 see all posts this is a mirror of the post at https://medium.com/@vitalikbuterin/a-proof-of-stake-design-philosophy-506585978d51 systems like ethereum (and bitcoin, and nxt, and bitshares, etc) are a fundamentally new class of cryptoeconomic organisms — decentralized, jurisdictionless entities that exist entirely in cyberspace, maintained by a combination of cryptography, economics and social consensus. they are kind of like bittorrent, but they are also not like bittorrent, as bittorrent has no concept of state — a distinction that turns out to be crucially important. they are sometimes described as decentralized autonomous corporations, but they are also not quite corporations — you can't hard fork microsoft. they are kind of like open source software projects, but they are not quite that either — you can fork a blockchain, but not quite as easily as you can fork openoffice. these cryptoeconomic networks come in many flavors — asic-based pow, gpu-based pow, naive pos, delegated pos, hopefully soon casper pos — and each of these flavors inevitably comes with its own underlying philosophy. one well-known example is the maximalist vision of proof of work, where "the" correct blockchain, singular, is defined as the chain that miners have burned the largest amount of economic capital to create. originally a mere in-protocol fork choice rule, this mechanism has in many cases been elevated to a sacred tenet — see this twitter discussion between myself and chris derose for an example of someone seriously trying to defend the idea in a pure form, even in the face of hash-algorithm-changing protocol hard forks. bitshares' delegated proof of stake presents another coherent philosophy, where everything once again flows from a single tenet, but one that can be described even more simply: shareholders vote. each of these philosophies; nakamoto consensus, social consensus, shareholder voting consensus, leads to its own set of conclusions and leads to a system of values that makes quite a bit of sense when viewed on its own terms — though they can certainly be criticized when compared against each other. casper consensus has a philosophical underpinning too, though one that has so far not been as succinctly articulated. myself, vlad, dominic, jae and others all have their own views on why proof of stake protocols exist and how to design them, but here i intend to explain where i personally am coming from. i'll proceed to listing observations and then conclusions directly. cryptography is truly special in the 21st century because cryptography is one of the very few fields where adversarial conflict continues to heavily favor the defender. castles are far easier to destroy than build, islands are defendable but can still be attacked, but an average person's ecc keys are secure enough to resist even state-level actors. cypherpunk philosophy is fundamentally about leveraging this precious asymmetry to create a world that better preserves the autonomy of the individual, and cryptoeconomics is to some extent an extension of that, except this time protecting the safety and liveness of complex systems of coordination and collaboration, rather than simply the integrity and confidentiality of private messages. systems that consider themselves ideological heirs to the cypherpunk spirit should maintain this basic property, and be much more expensive to destroy or disrupt than they are to use and maintain. the "cypherpunk spirit" isn't just about idealism; making systems that are easier to defend than they are to attack is also simply sound engineering. on medium to long time scales, humans are quite good at consensus. even if an adversary had access to unlimited hashing power, and came out with a 51% attack of any major blockchain that reverted even the last month of history, convincing the community that this chain is legitimate is much harder than just outrunning the main chain's hashpower. they would need to subvert block explorers, every trusted member in the community, the new york times, archive.org, and many other sources on the internet; all in all, convincing the world that the new attack chain is the one that came first in the information technology-dense 21st century is about as hard as convincing the world that the us moon landings never happened. these social considerations are what ultimately protect any blockchain in the long term, regardless of whether or not the blockchain's community admits it (note that bitcoin core does admit this primacy of the social layer). however, a blockchain protected by social consensus alone would be far too inefficient and slow, and too easy for disagreements to continue without end (though despite all difficulties, it has happened); hence, economic consensus serves an extremely important role in protecting liveness and safety properties in the short term. because proof of work security can only come from block rewards (in dominic williams' terms, it lacks two of the three es), and incentives to miners can only come from the risk of them losing their future block rewards, proof of work necessarily operates on a logic of massive power incentivized into existence by massive rewards. recovery from attacks in pow is very hard: the first time it happens, you can hard fork to change the pow and thereby render the attacker's asics useless, but the second time you no longer have that option, and so the attacker can attack again and again. hence, the size of the mining network has to be so large that attacks are inconceivable. attackers of size less than x are discouraged from appearing by having the network constantly spend x every single day. i reject this logic because (i) it kills trees, and (ii) it fails to realize the cypherpunk spirit — cost of attack and cost of defense are at a 1:1 ratio, so there is no defender's advantage. proof of stake breaks this symmetry by relying not on rewards for security, but rather penalties. validators put money ("deposits") at stake, are rewarded slightly to compensate them for locking up their capital and maintaining nodes and taking extra precaution to ensure their private key safety, but the bulk of the cost of reverting transactions comes from penalties that are hundreds or thousands of times larger than the rewards that they got in the meantime. the "one-sentence philosophy" of proof of stake is thus not "security comes from burning energy", but rather "security comes from putting up economic value-at-loss". a given block or state has $x security if you can prove that achieving an equal level of finalization for any conflicting block or state cannot be accomplished unless malicious nodes complicit in an attempt to make the switch pay $x worth of in-protocol penalties. theoretically, a majority collusion of validators may take over a proof of stake chain, and start acting maliciously. however, (i) through clever protocol design, their ability to earn extra profits through such manipulation can be limited as much as possible, and more importantly (ii) if they try to prevent new validators from joining, or execute 51% attacks, then the community can simply coordinate a hard fork and delete the offending validators' deposits. a successful attack may cost $50 million, but the process of cleaning up the consequences will not be that much more onerous than the geth/parity consensus failure of 2016.11.25. two days later, the blockchain and community are back on track, attackers are $50 million poorer, and the rest of the community is likely richer since the attack will have caused the value of the token to go up due to the ensuing supply crunch. that's attack/defense asymmetry for you. the above should not be taken to mean that unscheduled hard forks will become a regular occurrence; if desired, the cost of a single 51% attack on proof of stake can certainly be set to be as high as the cost of a permanent 51% attack on proof of work, and the sheer cost and ineffectiveness of an attack should ensure that it is almost never attempted in practice. economics is not everything. individual actors may be motivated by extra-protocol motives, they may get hacked, they may get kidnapped, or they may simply get drunk and decide to wreck the blockchain one day and to hell with the cost. furthermore, on the bright side, individuals' moral forbearances and communication inefficiencies will often raise the cost of an attack to levels much higher than the nominal protocol-defined value-at-loss. this is an advantage that we cannot rely on, but at the same time it is an advantage that we should not needlessly throw away. hence, the best protocols are protocols that work well under a variety of models and assumptions — economic rationality with coordinated choice, economic rationality with individual choice, simple fault tolerance, byzantine fault tolerance (ideally both the adaptive and non-adaptive adversary variants), ariely/kahneman-inspired behavioral economic models ("we all cheat just a little") and ideally any other model that's realistic and practical to reason about. it is important to have both layers of defense: economic incentives to discourage centralized cartels from acting anti-socially, and anti-centralization incentives to discourage cartels from forming in the first place. consensus protocols that work as-fast-as-possible have risks and should be approached very carefully if at all, because if the possibility to be very fast is tied to incentives to do so, the combination will reward very high and systemic-risk-inducing levels of network-level centralization (eg. all validators running from the same hosting provider). consensus protocols that don't care too much how fast a validator sends a message, as long as they do so within some acceptably long time interval (eg. 4–8 seconds, as we empirically know that latency in ethereum is usually ~500ms-1s) do not have these concerns. a possible middle ground is creating protocols that can work very quickly, but where mechanics similar to ethereum's uncle mechanism ensure that the marginal reward for a node increasing its degree of network connectivity beyond some easily attainable point is fairly low. from here, there are of course many details and many ways to diverge on the details, but the above are the core principles that at least my version of casper is based on. from here, we can certainly debate tradeoffs between competing values . do we give eth a 1% annual issuance rate and get an $50 million cost of forcing a remedial hard fork, or a zero annual issuance rate and get a $5 million cost of forcing a remedial hard fork? when do we increase a protocol's security under the economic model in exchange for decreasing its security under a fault tolerance model? do we care more about having a predictable level of security or a predictable level of issuance? these are all questions for another post, and the various ways of implementing the different tradeoffs between these values are questions for yet more posts. but we'll get to it :) dark mode toggle on radical markets 2018 apr 20 see all posts recently i had the fortune to have received an advance copy of eric posner and glen weyl's new book, radical markets, which could be best described as an interesting new way of looking at the subject that is sometimes called "political economy" tackling the big questions of how markets and politics and society intersect. the general philosophy of the book, as i interpret it, can be expressed as follows. markets are great, and price mechanisms are an awesome way of guiding the use of resources in society and bringing together many participants' objectives and information into a coherent whole. however, markets are socially constructed because they depend on property rights that are socially constructed, and there are many different ways that markets and property rights can be constructed, some of which are unexplored and potentially far better than what we have today. contra doctrinaire libertarians, freedom is a high-dimensional design space. the book interests me for multiple reasons. first, although i spend most of my time in the blockchain/crypto space heading up the ethereum project and in some cases providing various kinds of support to projects in the space, i do also have broader interests, of which the use of economics and mechanism design to make more open, free, egalitarian and efficient systems for human cooperation, including improving or replacing present-day corporations and governments, is a major one. the intersection of interests between the ethereum community and posner and weyl's work is multifaceted and plentiful; radical markets dedicates an entire chapter to the idea of "markets for personal data", redefining the economic relationship between ourselves and services like facebook, and well, look what the ethereum community is working on: markets for personal data. second, blockchains may well be used as a technical backbone for some of the solutions described in the book, and ethereum-style smart contracts are ideal for the kinds of complex systems of property rights that the book explores. third, the economic ideas and challenges that the book brings up are ideas that have also been explored, and will be continue to be explored, at great length by the blockchain community for its own purposes. posner and weyl's ideas often have the feature that they allow economic incentive alignment to serve as a substitute for subjective ad-hoc bureaucracy (eg. harberger taxes can essentially replace eminent domain), and given that blockchains lack access to trusted human-controlled courts, these kinds of solutions may prove to be be even more ideal for blockchain-based markets than they are for "real life". i will warn that readers are not at all guaranteed to find the book's proposals acceptable; at least the first three have already been highly controversial and they do contravene many people's moral preconceptions about how property should and should work and where money and markets can and can't be used. the authors are no strangers to controversy; posner has on previous occasions even proven willing to argue against such notions as human rights law. that said, the book does go to considerable lengths to explain why each proposal improves efficiency if it could be done, and offer multiple versions of each proposal in the hopes that there is at least one (even if partial) implementation of each idea that any given reader can find agreeable. what do posner and weyl talk about? the book is split into five major sections, each arguing for a particular reform: self-assessed property taxes, quadratic voting, a new kind of immigration program, breaking up big financial conglomerates that currently make banks and other industries act like monopolies even if they appear at first glance to be competitive, and markets for selling personal data. properly summarizing all five sections and doing them justice would take too long, so i will focus on a deep summary of one specific section, dealing with a new kind of property taxation, to give the reader a feel for the kinds of ideas that the book is about. harberger taxes see also: "property is only another name for monopoly", posner and weyl markets and private property are two ideas that are often considered together, and it is difficult in modern discourse to imagine one without (or even with much less of) the other. in the 19th century, however, many economists in europe were both libertarian and egalitarian, and it was quite common to appreciate markets while maintaining skepticism toward the excesses of private property. a rather interesting example of this is the bastiat-proudhon debate from 1849-1850 where the two dispute the legitimacy of charging interest on loans, with one side focusing on the mutual gains from voluntary contracts and the other focusing on their suspicion of the potential for people with capital to get even richer without working, leading to unbalanced capital accumulation. as it turns out, it is absolutely possible to have a system that contains markets but not property rights: at the end of every year, collect every piece of property, and at the start of the next year have the government auction every piece out to the highest bidder. this kind of system is intuitively quite unrealistic and impractical, but it has the benefit that it achieves perfect allocative efficiency: every year, every object goes to the person who can derive the most value from it (ie. the highest bidder). it also gives the government a large amount of revenue that could be used to completely substitute income and sales taxes or fund a basic income. now you might ask: doesn't the existing property system also achieve allocative efficiency? after all, if i have an apple, and i value it at $2, and you value it at $3, then you could offer me $2.50 and i would accept. however, this fails to take into account imperfect information: how do you know that i value it at $2, and not $2.70? you could offer to buy it for $2.99 so that you can be sure that you'll get it if you really are the one who values the apple more, but then you would be gaining practically nothing from the transaction. and if you ask me to set the price, how do i know that you value it at $3, and not $2.30? and if i set the price to $2.01 to be sure, i would be gaining practically nothing from the transaction. unfortunately, there is a result known as the myerson-satterthwaite theorem which means that no solution is efficient; that is, any bargaining algorithm in such a situation must at least sometimes lead to inefficiency from mutually beneficial deals falling through. if there are many buyers you have to negotiate with, things get even harder. if a developer (in the real estate sense) is trying to make a large project that requires buying 100 existing properties, and 99 have already agreed, the remaining one has a strong incentive to charge a very high price, much higher than their actual personal valuation of the property, hoping that the developer will have no choice but to pay up. well, not necessarily no choice. but a very inconvenient and both privately and socially wasteful choice. re-auctioning everything once a year completely solves this problem of allocative efficiency, but at a very high cost to investment efficiency: there's no point in building a house in the first place if six months later it will get taken away from you and re-sold in an auction. all property taxes have this problem; if building a house costs you $90 and brings you $100 of benefit, but then you have to pay $15 more property tax if you build the house, then you will not build the house and that $10 gain is lost to society. one of the more interesting ideas from the 19th century economists, and specifically henry george, was a kind of property tax that did not have this problem: the land value tax. the idea is to charge tax on the value of land, but not the improvements to the land; if you own a $100,000 plot of dirt you would have to pay $5,000 per year taxes on it regardless of whether you used the land to build a condominium or simply as a place to walk your pet doge. a doge. weyl and posner are not convinced that georgian land taxes are viable in practice: consider, for example, the empire state building. what is the pure value of the land beneath it? one could try to infer its value by comparing it to the value of adjoining land. but the building itself defines the neighborhood around it; removing the building would almost certainly change the value of its surrounding land. the land and the building, even the neighborhood, are so tied together, it would be hard to figure out a separate value for each of them. arguably this does not exclude the possibility of a different kind of georgian-style land tax: a tax based on the average of property values across a sufficiently large area. that would preserve the property that improving a single piece of land would not (greatly) perversely increase the taxes that they have to pay, without having to find a way to distinguish land from improvements in an absolute sense. but in any case, posner and weyl move on to their main proposal: self-assessed property taxes. consider a system where property owners themselves specify what the value of their property is, and pay a tax rate of, say, 2% of that value per year. but here is the twist: whatever value they specify for their property, they have to be willing to sell it to anyone at that price. if the tax rate is equal to the chance per year that the property gets sold, then this achieves optimal allocative efficiency: raising your self-assessed property value by $1 increases the tax you pay by $0.02, but it also means there is a 2% chance that someone will buy the property and pay $1 more, so there is no incentive to cheat in either direction. it does harm investment efficiency, but vastly less so than all property being re-auctioned every year. posner and weyl then point out that if more investment efficiency is desired, a hybrid solution with a lower property tax is possible: when the tax is reduced incrementally to improve investment efficiency, the loss in allocative efficiency is less than the gain in investment efficiency. the reason is that the most valuable sales are ones where the buyer is willing to pay significantly more than the seller is willing to accept. these transactions are the first ones enabled by a reduction in the price as even a small price reduction will avoid blocking these most valuable transactions. in fact, it can be shown that the size of the social loss from monopoly power grows quadratically in the extent of this power. thus, reducing the markup by a third eliminates close to \(\frac{5}{9} = (3^2-2^2)/(3^2\)) of the allocative harm from private ownership. this concept of quadratic deadweight loss is a truly important insight in economics, and is arguably the deep reason why "moderation in all things" is such an attractive principle: the first step you take away from an extreme will generally be the most valuable. the book then proceeds to give a series of side benefits that this tax would have, as well as some downsides. one interesting side benefit is that it removes an information asymmetry flaw that exists with property sales today, where owners have the incentive to expend effort on making their property look good even in potentially misleading ways. with a properly set harberger tax, if you somehow mange to trick the world into thinking your house is 5% more valuable, you'll get 5% more when you sell it but until that point you'll have to pay 5% more in taxes, or else someone will much more quickly snap it up from you at the original price. the downsides are smaller than they seem; for example, one natural disadvantage is that it exposes property owners to uncertainty due to the possibility that someone will snap up their property at any time, but that is hardly an unknown as it's a risk that renters already face every day. but weyl and posner do propose more moderate ways of introducing the tax that don't have these issues. first, the tax can be applied to types of property that are currently government owned; it's a potentially superior alternative to both continued government ownership and traditional full-on privatization. second, the tax can be applied to forms of property that are already "industrial" in usage: radio spectrum licenses, domain names, intellectual property, etc. the rest of the book the remaining chapters bring up similar ideas that are similar in spirit to the discussion on harberger taxes in their use of modern game-theoretic principles to make mathematically optimized versions of existing social institutions. one of the proposals is for something called quadratic voting, which i summarize as follows. suppose that you can vote as many times as you want, but voting costs "voting tokens" (say each citizen is assigned \(n\) voting tokens per year), and it costs tokens in a nonlinear way: your first vote costs one token, your second vote costs two tokens, and so forth. if someone feels more strongly about something, the argument goes, they would be willing to pay more for a single vote; quadratic voting takes advantage of this by perfectly aligning quantity of votes with cost of votes: if you're willing to pay up to 15 tokens for a vote, then you will keep buying votes until your last one costs 15 tokens, and so you will cast 15 votes in total. if you're willing to pay up to 30 tokens for a vote, then you will keep buying votes until you can't buy any more for a price less than or equal to 30 tokens, and so you will end up casting 30 votes. the voting is "quadratic" because the total amount you pay for \(n\) votes goes up proportionately to \(n^2\). after this, the book describes a market for immigration visas that could greatly expand the number of immigrants admitted while making sure local residents benefit and at the same time aligning incentives to encourage visa sponsors to choose immigrants that are more ikely to succeed in the country and less likely to commit crimes, then an enhancement to antitrust law, and finally the idea of setting up markets for personal data. markets in everything there are plenty of ways that one could respond to each individual proposal made in the book. i personally, for example, find the immigration visa scheme that posner and weyl propose well-intentioned and see how it could improve on the status quo, but also overcomplicated, and it seems simpler to me to have a scheme where visas are auctioned or sold every year, with an additional requirement for migrants to obtain liability insurance. robin hanson recently proposed greatly expanding liability insurance mandates as an alternative to many kinds of regulation, and while imposing new mandates on an entire society seems unrealistic, a new expanded immigration program seems like the perfect place to start considering them. paying people for personal data is interesting, but there are concerns about adverse selection: to put it politely, the kinds of people that are willing to sit around submitting lots of data to facebook all year to earn $16.92 (facebook's current annualized revenue per user) are not the kinds of people that advertisers are willing to burn hundreds of dollars per person trying to market rolexes and lambos to. however, what i find more interesting is the general principle that the book tries to promote. over the last hundred years, there truly has been a large amount of research into designing economic mechanisms that have desirable properties and that outperform simple two-sided buy-and-sell markets. some of this research has been put into use in some specific industries; for example, combinatorial auctions are used in airports, radio spectrum auctions and several other industrial use cases, but it hasn't really seeped into any kind of broader policy design; the political systems and property rights that we have are still largely the same as we had two centuries ago. so can we use modern economic insights to reform base-layer markets and politics in such a deep way, and if so, should we? normally, i love markets and clean incentive alignment, and dislike politics and bureaucrats and ugly hacks, and i love economics, and i so love the idea of using economic insights to design markets that work better so that we can reduce the role of politics and bureaucrats and ugly hacks in society. hence, naturally, i love this vision. so let me be a good intellectual citizen and do my best to try to make a case against it. there is a limit to how complex economic incentive structures and markets can be because there is a limit to users' ability to think and re-evaluate and give ongoing precise measurements for their valuations of things, and people value reliability and certainty. quoting steve waldman criticizing uber surge pricing: finally, we need to consider questions of economic calculation. in macroeconomics, we sometimes face tradeoffs between an increasing and unpredictably variable price-level and full employment. wisely or not, our current policy is to stabilize the price level, even at short-term cost to output and employment, because stable prices enable longer-term economic calculation. that vague good, not visible on a supply/demand diagram, is deemed worth very large sacrifices. the same concern exists in a microeconomic context. if the "ride-sharing revolution" really takes hold, a lot of us will have decisions to make about whether to own a car or rely upon the sidecars, lyfts, and ubers of the world to take us to work every day. to make those calculations, we will need something like predictable pricing. commuting to our minimum wage jobs (average is over!) by uber may be ok at standard pricing, but not so ok on a surge. in the desperate utopia of the "free-market economist", there is always a solution to this problem. we can define futures markets on uber trips, and so hedge our exposure to price volatility! in practice that is not so likely... and: it's clear that in a lot of contexts, people have a strong preference for price-predictability over immediate access. the vast majority of services that we purchase and consume are not price-rationed in any fine-grained way. if your hairdresser or auto mechanic is busy, you get penciled in for next week... strong property rights are valuable for the same reason: beyond the arguments about allocative and investment efficiency, they provide the mental convenience and planning benefits of predictability. it's worth noting that even uber itself doesn't do surge pricing in the "market-based" way that economists would recommend. uber is not a market where drivers can set their own prices, riders can see what prices are available, and themselves choose their tradeoff between price and waiting time. why does uber not do this? one argument is that, as steve waldman says, "uber itself is a cartel", and wants to have the power to adjust market prices not just for efficiency but also reasons such as profit maximization, strategically setting prices to drive out competing platforms (and taxis and public transit), and public relations. as waldman further points out, one uber competitor, sidecar, does have the ability for drivers to set prices, and i would add that i have seen ride-sharing apps in china where passengers can offer drivers higher prices to try to coax them to get a car faster. a possible counter-argument that uber might give is that drivers themselves are actually less good at setting optimal prices than uber's own algorithms, and in general people value the convenience of one-click interfaces over the mental complexity of thinking about prices. if we assume that uber won its market dominance over competitors like sidecar fairly, then the market itself has decided that the economic gain from marketizing more things is not worth the mental transaction costs. harberger taxes, at least to me, seem like they would lead to these exact kinds of issues multipled by ten; people are not experts at property valuation, and would have to spend a significant amount of time and mental effort figuring out what self-assessed value to put for their house, and they would complain much more if they accidentally put a value that's too low and suddenly find that their house is gone. if harberger taxes were to be applied to smaller property items as well, people would need to juggle a large amount of mental valuations of everything. a similar critique could apply to many kinds of personal data markets, and possibly even to quadratic voting if implemented in its full form. i could challenge this by saying "ah, even if that's true, this is the 21st century, we could have companies that build ais that make pricing decisions on your behalf, and people could choose the ai that seems to work best; there could even be a public option"; and posner and weyl themselves suggest that this is likely the way to go. and this is where the interesting conversation starts. tales from crypto land one reason why this discussion particularly interests me is that the cryptocurrency and blockchain space itself has, in some cases, run up against similar challenges. in the case of harberger taxes, we actually did consider almost exactly that same proposal in the context of the ethereum name system (our decentralized alternative to dns), but the proposal was ultimately rejected. i asked the ens developers why it was rejected. paraphrasing their reply, the challenge is as follows. many ens domain names are of a type that would only be interesting to precisely two classes of actors: (i) the "legitimate owner" of some given name, and (ii) scammers. furthermore, in some particular cases, the legitimate owner is uniquely underfunded, and scammers are uniquely dangerous. one particular case is myetherwallet, an ethereum wallet provider. myetherwallet provides an important public good to the ethereum ecosystem, making ethereum easier to use for many thousands of people, but is able to capture only a very small portion of the value that it provides; as a result, the budget that it has for outbidding others for the domain name is low. if a scammer gets their hands on the domain, users trusting myetherwallet could easily be tricked into sending all of their ether (or other ethereum assets) to a scammer. hence, because there is generally one clear "legitimate owner" for any domain name, a pure property rights regime presents little allocative efficiency loss, and there is a strong overriding public interest toward stability of reference (ie. a domain that's legitimate one day doesn't redirect to a scam the next day), so any level of harberger taxation may well bring more harm than good. i suggested to the ens developers the idea of applying harberger taxes to short domains (eg. abc.eth), but not long ones; the reply was that it would be too complicated to have two classes of names. that said, perhaps there is some version of the proposal that could satisfy the specific constraints here; i would be interested to hear posner and weyl's feedback on this particular application. another story from the blockchain and ethereum space that has a more pro-radical-market conclusion is that of transaction fees. the notion of mental transaction costs, the idea that the inconvenience of even thinking about whether or not some small payment for a given digital good is worth it is enough of a burden to prevent "micro-markets" from working, is often used as an argument for why mass adoption of blockchain tech would be difficult: every transaction requires a small fee, and the mental expenditure of figuring out what fee to pay is itself a major usability barrier. these arguments increased further at the end of last year, when both bitcoin and ethereum transaction fees briefly spiked up by a factor of over 100 due to high usage (talk about surge pricing!), and those who accidentally did not pay high enough fees saw their transactions get stuck for days. that said, this is a problem that we have now, arguably, to a large extent overcome. after the spikes at the end of last year, ethereum wallets developed more advanced algorithms for choosing what transaction fees to pay to ensure that one's transaction gets included in the chain, and today most users are happy to simply defer to them. in my own personal experience, the mental transaction costs of worrying about transaction fees do not really exist, much like a driver of a car does not worry about the gasoline consumed by every single turn, acceleration and braking made by their car. personal price-setting ais for interacting with open markets: already a reality in the ethereum transaction fee market a third kind of "radical market" that we are considering implementing in the context of ethereum's consensus system is one for incentivizing deconcentration of validator nodes in proof of stake consensus. it's important for blockchains to be decentralized, a similar challenge to what antitrust law tries to solve, but the tools at our disposal are different. posner and weyl's solution to antitrust, banning institutional investment funds from owning shares in multiple competitors in the same industry, is far too subjective and human-judgement-dependent to work in a blockchain, but for our specific context we have a different solution: if a validator node commits an error, it gets penalized an amount proportional to the number of other nodes that have committed an error around the same time. this incentivizes nodes to set themselves up in such a way that their failure rate is maximally uncorrelated with everyone else's failure rate, reducing the chance that many nodes fail at the same time and threaten to the blockchain's integrity. i want to ask posner and weyl: though our exact approach is fairly application-specific, could a similarly elegant "market-based" solution be discovered to incentivize market deconcentration in general? all in all, i am optimistic that the various behavioral kinks around implementing "radical markets" in practice could be worked out with the help of good defaults and personal ais, though i do think that if this vision is to be pushed forward, the greatest challenge will be finding progressively larger and more meaningful places to test it out and show that the model works. i particularly welcome the use of the blockchain and crypto space as a testing ground. another kind of radical market the book as a whole tends to focus on centralized reforms that could be implemented on an economy from the top down, even if their intended long-term effect is to push more decision-making power to individuals. the proposals involve large-scale restructurings of how property rights work, how voting works, how immigration and antitrust law works, and how individuals see their relationship with property, money, prices and society. but there is also the potential to use economics and game theory to come up with decentralized economic institutions that could be adopted by smaller groups of people at a time. perhaps the most famous examples of decentralized institutions from game theory and economics land are (i) assurance contracts, and (ii) prediction markets. an assurance contract is a system where some public good is funded by giving anyone the opportunity to pledge money, and only collecting the pledges if the total amount pledged exceeds some threshold. this ensures that people can donate money knowing that either they will get their money back or there actually will be enough to achieve some objective. a possible extension of this concept is alex tabarrok's dominant assurance contracts, where an entrepreneur offers to refund participants more than 100% of their deposits if a given assurance contract does not raise enough money. prediction markets allow people to bet on the probability that events will happen, potentially even conditional on some action being taken ("i bet $20 that unemployment will go down if candidate x wins the election"); there are techniques for people interested in the information to subsidize the markets. any attempt to manipulate the probability that a prediction market shows simply creates an opportunity for people to earn free money (yes i know, risk aversion and capital efficiency etc etc; still close to free) by betting against the manipulator. posner and weyl do give one example of what i would call a decentralized institution: a game for choosing who gets an asset in the event of a divorce or a company splitting in half, where both sides provide their own valuation, the person with the higher valuation gets the item, but they must then give an amount equal to half the average of the two valuations to the loser. there's some economic reasoning by which this solution, while not perfect, is still close to mathematically optimal. one particular category of decentralized institutions i've been interested in is improving incentivization for content posting and content curation in social media. some ideas that i have had include: proof of stake conditional hashcash (when you send someone an email, you give them the opportunity to burn $0.5 of your money if they think it's spam) prediction markets for content curation (use prediction markets to predict the results of a moderation vote on content, thereby encouraging a market of fast content pre-moderators while penalizing manipulative pre-moderation) conditional payments for paywalled content (after you pay for a piece of downloadable content and view it, you can decide after the fact if payments should go to the author or to proportionately refund previous readers) and ideas i have had in other contexts: call-out assurance contracts daicos (a more decentralized and safer alternative to icos) twitter scammers: can prediction markets incentivize an autonomous swarm of human and ai-driven moderators to flag these posts and warn users not to send them ether within a few seconds of the post being made? and could such a system be generalized to the entire internet, where these is no single centralized moderator that can easily take posts down? some ideas others have had for decentralized institutions in general include: trustdavis (adding skin-in-the-game to e-commerce reputations by making e-commerce ratings be offers to insure others against the receiver of the rating committing fraud) circles (decentralized basic income through locally fungible coin issuance) markets for captcha services digitized peer to peer rotating savings and credit associations token curated registries crowdsourced smart contract truth oracles using blockchain-based smart contracts to coordinate unions i would be interested in hearing posner and weyl's opinion on these kinds of "radical markets", that groups of people can spin up and start using by themselves without requiring potentially contentious society-wide changes to political and property rights. could decentralized institutions like these be used to solve the key defining challenges of the twenty first century: promoting beneficial scientific progress, developing informational public goods, reducing global wealth inequality, and the big meta-problem behind fake news, government-driven and corporate-driven social media censorship, and regulation of cryptocurrency products: how do we do quality assurance in an open society? all in all, i highly recommend radical markets (and by the way i also recommend eliezer yudkowsky's inadequate equilibria) to anyone interested in these kinds of issues, and look forward to seeing the discussion that the book generates. on bitcoin maximalism, and currency and platform network effects | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on bitcoin maximalism, and currency and platform network effects posted by vitalik buterin on november 20, 2014 research & development one of the latest ideas that has come to recently achieve some prominence in parts of the bitcoin community is the line of thinking that has been described by both myself and others as "bitcoin dominance maximalism" or just "bitcoin maximalism" for short essentially, the idea that an environment of multiple competing cryptocurrencies is undesirable, that it is wrong to launch "yet another coin", and that it is both righteous and inevitable that the bitcoin currency comes to take a monopoly position in the cryptocurrency scene. note that this is distinct from a simple desire to support bitcoin and make it better; such motivations are unquestionably beneficial and i personally continue to contribute to bitcoin regularly via my python library pybitcointools. rather, it is a stance that building something on bitcoin is the only correct way to do things, and that doing anything else is unethical (see this post for a rather hostile example). bitcoin maximalists often use "network effects" as an argument, and claim that it is futile to fight against them. however, is this ideology actually such a good thing for the cryptocurrency community? and is its core claim, that network effects are a powerful force strongly favoring the eventual dominance of already established currencies, really correct, and even if it is, does that argument actually lead where its adherents think it leads? the technicals first, an introduction to the technical strategies at hand. in general, there are three approaches to creating a new crypto protocol: build on bitcoin the blockchain, but not bitcoin the currency (metacoins, eg. most features of counterparty) build on bitcoin the currency, but not bitcoin the blockchain (sidechains) create a completely standalone platform meta-protocols are relatively simple to describe: they are protocols that assign a secondary meaning to certain kinds of specially formatted bitcoin transactions, and the current state of the meta-protocol can be determined by scanning the blockchain for valid metacoin transactions and sequentially processing the valid ones. the earliest meta-protocol to exist was mastercoin; counterparty is a newer one. meta-protocols make it much quicker to develop a new protocol, and allow protocols to benefit directly from bitcoin's blockchain security, although at a high cost: meta-protocols are not compatible with light client protocols, so the only efficient way to use a meta-protocol is via a trusted intermediary. sidechains are somewhat more complicated. the core underlying idea revolves around a "two-way-pegging" mechanism, where a "parent chain" (usually bitcoin) and a "sidechain" share a common currency by making a unit of one convertible into a unit of the other. the way it works is as follows. first, in order to get a unit of side-coin, a user must send a unit of parent-coin into a special "lockbox script", and then submit a cryptographic proof that this transaction took place into the sidechain. once this transaction confirms, the user has the side-coin, and can send it at will. when any user holding a unit of side-coin wants to convert it back into parent-coin, they simply need to destroy the side-coin, and then submit a proof that this transaction took place to a lockbox script on the main chain. the lockbox script would then verify the proof, and if everything checks out it would unlock the parent-coin for the submitter of the side-coin-destroying transaction to spend. unfortunately, it is not practical to use the bitcoin blockchain and currency at the same time; the basic technical reason is that nearly all interesting metacoins involve moving coins under more complex conditions than what the bitcoin protocol itself supports, and so a separate "coin" is required (eg. msc in mastercoin, xcp in counterparty). as we will see, each of these approaches has its own benefits, but it also has its own flaws. this point is important; particularly, note that many bitcoin maximalists' recent glee at counterparty forking ethereum was misplaced, as counterparty-based ethereum smart contracts cannot manipulate btc currency units, and the asset that they are instead likely to promote (and indeed already have promoted) is the xcp. network effects now, let us get to the primary argument at play here: network effects. in general, network effects can be defined simply: a network effect is a property of a system that makes the system intrinsically more valuable the more people use it. for example, a language has a strong network effect: esperanto, even if it is technically superior to english in the abstract, is less useful in practice because the whole point of a language is to communicate with other people and not many other people speak esperanto. on the other hand, a single road has a negative network effect: the more people use it the more congested it becomes. in order to properly understand what network effects are at play in the cryptoeconomic context, we need to understand exactly what these network effects are, and exactly what thing each effect is attached to. thus, to start off, let us list a few of the major ones (see here and here for primary sources): security effect: systems that are more widely adopted derive their consensus from larger consensus groups, making them more difficult to attack. payment system network effect: payment systems that are accepted by more merchants are more attractive to consumers, and payment systems used by more consumers are more attractive to merchants. developer network effect: there are more people interested in writing tools that work with platforms that are widely adopted, and the greater number of these tools will make the platform easier to use. integration network effect: third party platforms will be more willing to integrate with a platform that is widely adopted, and the greater number of these tools will make the platform easier to use. size stability effect: currencies with larger market cap tend to be more stable, and more established cryptocurrencies are seen as more likely (and therefore by self-fulfilling-prophecy actually are more likely) to remain at nonzero value far into the future. unit of account network effect: currencies that are very prominent, and stable, are used as a unit of account for pricing goods and services, and it is cognitively easier to keep track of one's funds in the same unit that prices are measured in. market depth effect: larger currencies have higher market depth on exchanges, allowing users to convert larger quantities of funds in and out of that currency without taking a hit on the market price. market spread effect: larger currencies have higher liquidity (ie. lower spread) on exchanges, allowing users to convert back and forth more efficiently. intrapersonal single-currency preference effect: users that already use a currency for one purpose prefer to use it for other purposes both due to lower cognitive costs and because they can maintain a lower total liquid balance among all cryptocurrencies without paying interchange fees. interpersonal single-currency preference effect: users prefer to use the same currency that others are using to avoid interchange fees when making ordinary transactions marketing network effect: things that are used by more people are more prominent and thus more likely to be seen by new users. additionally, users have more knowledge about more prominent systems and thus are less concerned that they might be exploited by unscrupulous parties selling them something harmful that they do not understand. regulatory legitimacy network effect: regulators are less likely to attack something if it is prominent because they will get more people angry by doing so the first thing that we see is that these network effects are actually rather neatly split up into several categories: blockchain-specific network effects (1), platform-specific network effects (2-4), currency-specific network effects (5-10), and general network effects (11-12), which are to a large extent public goods across the entire cryptocurrency industry. there is a substantial opportunity for confusion here, since bitcoin is simultaneously a blockchain, a currency and a platform, but it is important to make a sharp distinction between the three. the best way to delineate the difference is as follows: a currency is something which is used as a medium of exchange or store of value; for example, dollars, btc and doge. a platform is a set of interoperating tools and infrastructure that can be used to perform certain tasks; for currencies, the basic kind of platform is the collection of a payment network and the tools needed to send and receive transactions in that network, but other kinds of platforms may also emerge. a blockchain is a consensus-driven distributed database that modifies itself based on the content of valid transactions according to a set of specified rules; for example, the bitcoin blockchain, the litecoin blockchain, etc. to see how currencies and platforms are completely separate, the best example to use is the world of fiat currencies. credit cards, for example, are a highly multi-currency platform. someone with a credit card from canada tied to a bank account using canadian dollars can spend funds at a merchant in switzerland accepting swiss francs, and both sides barely know the difference. meanwhile, even though both are (or at least can be) based on the us dollar, cash and paypal are completely different platforms; a merchant accepting only cash will have a hard time with a customer who only has a paypal account. as for how platforms and blockchains are separate, the best example is the bitcoin payment protocol and proof of existence. although the two use the same blockchain, they are completely different applications, users of one have no idea how to interpret transactions associated with the other, and it is relatively easy to see how they benefit from completely different network effects so that one can easily catch on without the other. note that protocols like proof of existence and factom are mostly exempt from this discussion; their purpose is to embed hashes into the most secure available ledger, and while a better ledger has not materialized they should certainly use bitcoin, particularly because they can use merkle trees to compress a large number of proofs into a single hash in a single transaction. network effects and metacoins now, in this model, let us examine metacoins and sidechains separately. with metacoins, the situation is simple: metacoins are built on bitcoin the blockchain, and not bitcoin the platform or bitcoin the currency. to see the former, note that users need to download a whole new set of software packages in order to be able to process bitcoin transactions. there is a slight cognitive network effect from being able to use the same old infrastructure of bitcoin private/public key pairs and addresses, but this is a network effect for the combination of ecdsa, sha256+ripemd160 and base 58 and more generally the whole concept of cryptocurrency, not the bitcoin platform; dogecoin inherits exactly the same gains. to see the latter, note that, as mentioned above, counterparty has its own internal currency, the xcp. hence, metacoins benefit from the network effect of bitcoin's blockchain security, but do not automatically inherit all of the platform-specific and currency-specific network effects. of course, metacoins' departure from the bitcoin platform and bitcoin currency is not absolute. first of all, even though counterparty is not "on" the bitcoin platform, it can in a very meaningful sense be said to be "close" to the bitcoin platform one can exchange back and forth between btc and xcp very cheaply and efficiently. cross-chain centralized or decentralized exchange, while possible, is several times slower and more costly. second, some features of counterparty, particularly the token sale functionality, do not rely on moving currency units under any conditions that the bitcoin protocol does not support, and so one can use that functionality without ever purchasing xcp, using btc directly. finally, transaction fees in all metacoins can be paid in btc, so in the case of purely non-financial applications metacoins actually do fully benefit from bitcoin's currency effect, although we should note that in most non-financial cases developers are used to messaging being free, so convincing anyone to use a non-financial blockchain dapp at $0.05 per transaction will likely be an uphill battle. in some of these applications particularly, perhaps to bitcoin maximalists' chagrin, counterparty's crypto 2.0 token sales, the desire to move back and forth quickly to and from bitcoin, as well as the ability to use it directly, may indeed create a platform network effect that overcomes the loss of secure light client capability and potential for blockchain speed and scalability upgrades, and it is in these cases that metacoins may find their market niche. however, metacoins are most certainly not an all-purpose solution; it is absurd to believe that bitcoin full nodes will have the computational ability to process every single crypto transaction that anyone will ever want to do, and so eventually movement to either scalable architectures or multichain environments will be necessary. network effects and sidechains sidechains have the opposite properties of metacoins. they are built on bitcoin the currency, and thus benefit from bitcoin's currency network effects, but they are otherwise exactly identical to fully independent chains and have the same properties. this has several pros and cons. on the positive side, it means that, although "sidechains" by themselves are not a scalability solution as they do not solve the security problem, future advancements in multichain, sharding or other scalability strategies are all open to them to adopt. on the negative side, however, they do not benefit from bitcoin's platform network effects. one must download special software in order to be able to interact with a sidechain, and one must explicitly move one's bitcoins onto a sidechain in order to be able to use it a process wich is equally as difficult as converting them into a new currency in a new network via a decentralized exchange. in fact, blockstream employees have themselves admitted that the process for converting side-coins back into bitcoins is relatively inefficient, to the point that most people seeking to move their bitcoins there and back will in fact use exactly the same centralized or decentralized exchange processes as would be used to migrate to a different currency on an independent blockchain. additionally, note that there is one security approach that independent networks can use which is not open to sidechains: proof of stake. the reasons for this are twofold. first one of the key arguments in favor of proof of stake is that even a successful attack against proof of stake will be costly for the attacker, as the attacker will need to keep his currency units deposited and watch their value drop drastically as the market realizes that the coin is compromised. this incentive effect does not exist if the only currency inside of a network is pegged to an external asset whose value is not so closely tied to that network's success. second, proof of stake gains much of its security because the process of buying up 50% of a coin in order to mount a takeover attack will itself increase the coin's price drastically, making the attack even more expensive for the attacker. in a proof of stake sidechain, however, one can easily move a very large quantity of coins into a chain from the parent chain, an mount the attack without moving the asset price at all. note that both of these arguments continue to apply even if bitcoin itself upgrades to proof of stake for its security. hence, if you believe that proof of stake is the future, then both metacoins and sidechains (or at least pure sidechains) become highly suspect, and thus for that purely technical reason bitcoin maximalism (or, for that matter, ether maximalism, or any other kind of currency maximalism) becomes dead in the water. currency network effects, revisited altogether, the conclusion from the above two points is twofold. first, there is no universal and scalable approach that allows users to benefit from bitcoin's platform network effects. any software solution that makes it easy for bitcoin users to move their funds to sidechains can be easily converted into a solution that makes it just as easy for bitcoin users to convert their funds into an independent currency on an independent chain. on the other hand, however, currency network effects are another story, and may indeed prove to be a genuine advantage for bitcoin-based sidechains over fully independent networks. so, what exactly are these effects and how powerful is each one in this context? let us go through them again: size-stability network effect (larger currencies are more stable) this network effect is legitimate, and bitcoin has been shown to be less volatile than smaller coins. unit of account network effect (very large currencies become units of account, leading to more purchasing power stability via price stickiness as well as higher salience) unfortunately, bitcoin will likely never be stable enough to trigger this effect; the best empirical evidence we can see for this is likely the valuation history of gold. market depth effect (larger currencies support larger transactions without slippage and have a lower bid/ask spread) these effect are legitimate up to a point, but then beyond that point (perhaps a market cap of $10-$100m), the market depth is imply good enough and the spread is low enough for nearly all types of transactions, and the benefit from further gains is small. single-currency preference effect (people prefer to deal with fewer currencies, and prefer to use the same currencies that others are using) the intrapersonal and interpersonal parts to this effect are legitimate, but we note that (i) the intrapersonal effect only applies within individual people, not between people, so it does not prevent an ecosystem with multiple preferred global currencies from existing, and (ii) the interpersonal effect is small as interchange fees especially in crypto tend to be very low, less than 0.30%, and will likely go down to essentially zero with decentralized exchange. hence, the single-currency preference effect is likely the largest concern, followed by the size stability effects, whereas the market depth effects are likely relatively tiny once a cryptocurrency gets to a substantial size. however, it is important to note that the above points have several major caveats. first, if (1) and (2) dominate, then we know of explicit strategies for making a new coin that is even more stable than bitcoin even at a smaller size; thus, they are certainly not points in bitcoin's favor. second, those same strategies (particularly the exogenous ones) can actually be used to create a stable coin that is pegged to a currency that has vastly larger network effects than even bitcoin itself; namely, the us dollar. the us dollar is thousands of times larger than bitcoin, people are already used to thinking in terms of it, and most importantly of all it actually maintains its purchasing power at a reasonable rate in the short to medium term without massive volatility. employees of blockstream, the company behind sidechains, have often promoted sidechains under the slogan "innovation without speculation"; however, the slogan ignores that bitcoin itself is quite speculative and as we see from the experience of gold always will be, so seeking to install bitcoin as the only cryptoasset essentially forces all users of cryptoeconomic protocols to participate in speculation. want true innovation without speculation? then perhaps we should all engage in a little us dollar stablecoin maximalism instead. finally, in the case of transaction fees specifically, the intrapersonal single-currency preference effect arguably disappears completely. the reason is that the quantities involved are so small ($0.01-$0.05 per transaction) that a dapp can simply siphon off $1 from a user's bitcoin wallet at a time as needed, not even telling the user that other currencies exist, thereby lowering the cognitive cost of managing even thousands of currencies to zero. the fact that this token exchange is completely non-urgent also means that the client can even serve as a market maket while moving coins from one chain to the other, perhaps even earning a profit on the currency interchange bid/ask spread. furthermore, because the user does not see gains and losses, and the user's average balance is so low that the central limit theorem guarantees with overwhelming probability that the spikes and drops will mostly cancel each other out, stability is also fairly irrelevant. hence, we can make the point that alternative tokens which are meant to serve primarily as "cryptofuels" do not suffer from currency-specific network effect deficiencies at all. let a thousand cryptofuels bloom. incentive and psychological arguments there is another class of argument, one which may perhaps be called a network effect but not completely, for why a service that uses bitcoin as a currency will perform better: the incentivized marketing of the bitcoin community. the argument goes as follows. services and platforms based on bitcoin the currency (and to a slight extent services based on bitcoin the platform) increase the value of bitcoin. hence, bitcoin holders would personally benefit from the value of their btc going up if the service gets adopted, and are thus motivated to support it. this effect occurs on two levels: the individual and the corporate. the corporate effect is a simple matter of incentives; large businesses will actually support or even create bitcoin-based dapps to increase bitcoin's value, simply because they are so large that even the portion of the benefit that personally accrues to themselves is enough to offset the costs; this is the "speculative philanthropy" strategy described by daniel krawisz. the individual effect is not so much directly incentive-based; each individual's ability to affect bitcoin's value is tiny. rather, it's more a clever exploitation of psychological biases. it's well-known that people tend to change their moral values to align with their personal interests, so the channel here is more complex: people who hold btc start to see it as being in the common interest for bitcoin to succeed, and so they will genuinely and excitedly support such applications. as it turns out, even a small amount of incentive suffices to shift over people's moral values to such a large extent, creating a psychological mechanism that manages to overcome not just the coordination problem but also, to a weak extent, the public goods problem. there are several major counterarguments to this claim. first, it is not at all clear that the total effect of the incentive and psychological mechanisms actually increases as the currency gets larger. although a larger size leads to more people affected by the incentive, a smaller size creates a more concentrated incentive, as people actually have the opportunity to make a substantial difference to the success of the project. the tribal psychology behind incentive-driven moral adjustment may well be stronger for small "tribes" where individuals also have strong social connections to each other than larger tribes where such connections are more diffuse; this is somewhat similar to the gemeinschaft vs gesellschaft distinction in sociology. perhaps a new protocol needs to have a concentrated set of highly incentivized stakeholders in order to seed a community, and bitcoin maximalists are wrong to try to knock this ladder down after bitcoin has so beautifully and successfully climbed up it. in any case, all of the research around optimum currency areas will have to be heavily redone in the context of the newer volatile cryptocurrencies, and the results may well go down either way. second, the ability for a network to issue units of a new coin has been proven to be a highly effective and successful mechanism for solving the public goods problem of funding protocol development, and any platform that does not somehow take advantage of the seignorage revenue from creating a new coin is at a substantial disadvantage. so far, the only major crypto 2.0 protocol-building company that has successfully funded itself without some kind of "pre-mine" or "pre-sale" is blockstream (the company behind sidechains), which recently received $21 million of venture capital funding from silicon valley investors. given blockstream's self-inflicted inability to monetize via tokens, we are left with three viable explanations for how investors justified the funding: the funding was essentially an act of speculative philathropy on the part of silicon valley venture capitalists looking to increase the value of their btc and their other btc-related investments. blockstream intends to earn revenue by taking a cut of the fees from their blockchains (non-viable because the public will almost certainly reject such a clear and blatant centralized siphoning of resources even more virulently then they would reject a new currency) blockstream intends to "sell services", ie. follow the redhat model (viable for them but few others; note that the total room in the market for redhat-style companies is quite small) both (1) and (3) are highly problematic; (3) because it means that few other companies will be able to follow its trail and because it gives them the incentive to cripple their protocols so they can provide centralized overlays, and (1) because it means that crypto 2.0 companies must all follow the model of sucking up to the particular concentrated wealthy elite in silicon valley (or maybe an alternative concentrated wealthy elite in china), hardly a healthy dynamic for a decentralized ecosystem that prides itself on its high degree of political independence and its disruptive nature. ironically enough, the only "independent" sidechain project that has so far announced itself, truthcoin, has actually managed to get the best of both worlds: the project got on the good side of the bitcoin maximalist bandwagon by announcing that it will be a sidechain, but in fact the development team intends to introduce into the platform two "coins" one of which will be a btc sidechain token and the other an independent currency that is meant to be, that's right, crowd-sold. a new strategy thus, we see that while currency network effects are sometimes moderately strong, and they will indeed exert a preference pressure in favor of bitcoin over other existing cryptocurrencies, the creation of an ecosystem that uses bitcoin exclusively is a highly suspect endeavor, and one that will lead to a total reduction and increased centralization of funding (as only the ultra-rich have sufficient concentrated incentive to be speculative philanthropists), closed doors in security (no more proof of stake), and is not even necessarily guaranteed to end with bitcoin willing. so is there an alternative strategy that we can take? are there ways to get the best of both worlds, simultaneously currency network effects and securing the benefits of new protocols launching their own coins? as it turns out, there is: the dual-currency model. the dual-currency model, arguably pioneered by robert sams, although in various incarnations independently discovered by bitshares, truthcoin and myself, is at the core simple: every network will contain two (or even more) currencies, splitting up the role of medium of transaction and vehicle of speculation and stake (the latter two roles are best merged, because as mentioned above proof of stake works best when participants suffer the most from a fork). the transactional currency will be either a bitcoin sidechain, as in truthcoin's model, or an endogenous stablecoin, or an exogenous stablecoin that benefits from the almighty currency network effect of the us dollar (or euro or cny or sdr or whatever else). hayekian currency competition will determine which kind of bitcoin, altcoin or stablecoin users prefer; perhaps sidechain technology can even be used to make one particular stablecoin transferable across many networks. the vol-coin will be the unit of measurement of consensus, and vol-coins will sometimes be absorbed to issue new stablecoins when stablecoins are consumed to pay transaction fees; hence, as explainted in the argument in the linked article on stablecoins, vol-coins can be valued as a percentage of future transaction fees. vol-coins can be crowd-sold, maintaining the benefits of a crowd sale as a funding mechanism. if we decide that explicit pre-mines or pre-sales are "unfair", or that they have bad incentives because the developers' gain is frontloaded, then we can instead use voting (as in dpos) or prediction markets instead to distribute coins to developers in a decentralized way over time. another point to keep in mind is, what happens to the vol-coins themselves? technological innovation is rapid, and if each network gets unseated within a few years, then the vol-coins may well never see substantial market cap. one answer is to solve the problem by using a clever combination of satoshian thinking and good old-fashioned recursive punishment systems from the offline world: establish a social norm that every new coin should pre-allocate 50-75% of its units to some reasonable subset of the coins that came before it that directly inspired its creation, and enforce the norm blockchain-style if your coin does not honor its ancestors, then its descendants will refuse to honor it, instead sharing the extra revenues between the originally cheated ancestors and themselves, and no one will fault them for that. this would allow vol-coins to maintain continuity over the generations. bitcoin itself can be included among the list of ancestors for any new coin. perhaps an industry-wide agreement of this sort is what is needed to promote the kind of cooperative and friendly evolutionary competition that is required for a multichain cryptoeconomy to be truly successful. would we have used a vol-coin/stable-coin model for ethereum had such strategies been well-known six months ago? quite possibly yes; unfortunately it's too late to make the decision now at the protocol level, particularly since the ether genesis block distribution and supply model is essentially finalized. fortunately, however, ethereum allows users to create their own currencies inside of contracts, so it is entirely possible that such a system can simply be grafted on, albeit slightly unnaturally, over time. even without such a change, ether itself will retain a strong and steady value as a cryptofuel, and as a store of value for ethereum-based security deposits, simply because of the combination of the ethereum blockchain's network effect (which actually is a platform network effect, as all contracts on the ethereum blockchain have a common interface and can trivially talk to each other) and the weak-currency-network-effect argument described for cryptofuels above preserves for it a stable position. for 2.0 multichain interaction, however, and for future platforms like truthcoin, the decision of which new coin model to take is all too relevant. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle on path independence 2017 jun 22 see all posts suppose that someone walks up to you and starts exclaiming to you that he thinks he has figured out how to create a source of unlimited free energy. his scheme looks as follows. first, you get a spaceship up to low earth orbit. there, earth's gravity is fairly high, and so the spaceship will start to accelerate heavily toward the earth. the spaceship puts itself into a trajectory so that it barely brushes past the earth's atmosphere, and then keeps hurtling far into space. further in space, the gravity is lower, and so the spaceship can go higher before it starts once again coming down. when it comes down, it takes a curved path toward the earth, so as to maximize its time in low orbit, maximizing the acceleration it gets from the high gravity there, so that after it passes by the earth it goes even higher. after it goes high enough, it flies through the earth's atmosphere, slowing itself down but using the waste heat to power a thermal reactor. then, it would go back to step one and keep going. something like this: now, if you know anything about newtonian dynamics, chances are you'll immediately recognize that this scheme is total bollocks. but how do you know? you could make an appeal to symmetry, saying "look, for every slice of the orbital path where you say gravity gives you high acceleration, there's a corresponding slice of the orbital path where gravity gives you just as high deceleration, so i don't see where the net gains are coming from". but then, suppose the man presses you. "ah," he says, "but in that slice where there is high acceleration your initial velocity is low, and so you spend a lot of time inside of it, whereas in the corresponding slice, your incoming velocity is high, and so you have less time to decelerate". how do you really, conclusively, prove him wrong? one approach is to dig deeply into the math, calculate the integrals, and show that the supposed net gains are in fact exactly equal to zero. but there is also a simple approach: recognize that energy is path-independent. that is, when the spaceship moves from point \(a\) to point \(b\), where point \(b\) is closer to the earth, its kinetic energy certainly goes up because its speed increases. but because total energy (kinetic plus potential) is conserved, and potential energy is only dependent on the spaceship's position, and not how it got there, we know that regardless of what path from point \(a\) to point \(b\) the spaceship takes, once it gets to point \(b\) the total change in kinetic energy will be exactly the same. different paths, same change in energy furthermore, we know that the kinetic energy gain from going from point \(a\) to point \(a\) is also independent of the path you take along the way: in all cases it's exactly zero. one concern sometimes cited against on-chain market makers (that is, fully automated on-chain mechanisms that act as always-available counterparties for people who wish to trade one type of token for another) is that they are invariably easy to exploit. as an example, let me quote a recent post discussing this issue in the context of bancor: the prices that bancor offers for tokens have nothing to do with the actual market equilibrium. bancor will always trail the market, and in doing so, will bleed its reserves. a simple thought experiment suffices to illustrate the problem. suppose that market panic sets around x. unfounded news about your system overtake social media. let's suppose that people got convinced that your ceo has absconded to a remote island with no extradition treaty, that your cfo has been embezzling money, and your cto was buying drugs from the darknet markets and shipping them to his work address to make a scarface-like mound of white powder on his desk. worse, let's suppose that you know these allegations to be false. they were spread by a troll army wielded by a company with no products, whose business plan is to block everyone's coin stream. bancor would offer ever decreasing prices for x coins during a bank run, until it has no reserves left. you'd watch the market panic take hold and eat away your reserves. recall that people are convinced that the true value of x is 0 in this scenario, and the bancor formula is guaranteed to offer a price above that. so your entire reserve would be gone. the post discusses many issues around the bancor protocol, including details such as code quality, and i will not touch on any of those; instead, i will focus purely on the topic of on-chain market maker efficiency and exploitability, using bancor (along with mkr) purely as examples and not seeing to make any judgements on the quality of either project as a whole. for many classes of naively designed on-chain market makers, the comment above about exploitability and trailing markets applies verbatim, and quite seriously so. however, there are also classes of on-chain market makers that are definitely not suspect to their entire reserve being drained due to some kind of money-pumping attack. to take a simple example, consider the market maker selling mkr for eth whose internal state consists of a current price, \(p\), and which is willing to buy or sell an infinitesimal amount of mkr at each price level. for example, suppose that \(p = 5\), and you wanted to buy 2 mkr. the market would sell you: 0.00...01 mkr at a price of 5 eth/mkr 0.00...01 mkr at a price of 5.00...01 eth/mkr 0.00...01 mkr at a price of 5.00...02 eth/mkr .... 0.00...01 mkr at a price of 6.99...98 eth/mkr 0.00...01 mkr at a price of 6.99...99 eth/mkr altogether, it's selling you 2 mkr at an average price of 6 eth/mkr (ie. total cost 12 eth), and at the end of the operation \(p\) has increased to 7. if someone then wanted to sell 1 mkr, they would be spending 6.5 eth, and at the end of that operation \(p\) would drop back down to 6. now, suppose that i told you that such a market maker started off at a price of \(p = 5\), and after an unspecified series of events \(p\) is now 4. two questions: how much mkr did the market maker gain or lose? how much eth did the market maker gain or lose? the answers are: it gained 1 mkr, and lost 4.5 eth. notice that this result is totally independent of the path that \(p\) took. those answers are correct if \(p\) went from 5 to 4 directly with one buyer, they're correct if there was first one buyer that took \(p\) from 5 to 4.7 and a second buyer that took \(p\) the rest of the way to 4, and they're even correct if \(p\) first dropped to 2, then increased to 9.818, then dropped again to 0.53, then finally rose again to 4. why is this the case? the simplest way to see this is to see that if \(p\) drops below 4 and then comes back up to 4, the sells on the way down are exactly counterbalanced by buys on the way up; each sell has a corresponding buy of the same magnitude at the exact same price. but we can also see this by viewing the market maker's core mechanism differently. define the market maker as having a single-dimensional internal state \(p\), and having mkr and eth balances defined by the following formulas: \(mkr\_balance(p) = 10 p\) \(eth\_balance(p) = p^2/2\) anyone has the power to "edit" \(p\) (though only to values between 0 and 10), but they can only do so by supplying the right amount of mkr or eth, and getting the right amount of mkr and eth back, so that the balances still match up; that is, so that the amount of mkr and eth held by the market maker after the operation is the amount that they are supposed to hold according to the above formulas, with the new value for \(p\) that was set. any edit to \(p\) that does not come with mkr and eth transactions that make the balances match up automatically fails. now, the fact that any series of events that drops \(p\) from 5 to 4 also raises the market maker's mkr balance by 1 and drops its eth balance by 4.5, regardless of what series of events it was, should look elementary: \(mkr\_balance(4) mkr\_balance(5) = 1\) and \(eth\_balance(4) eth\_balance(5) = -4.5\). what this means is that a "reserve bleeding" attack on a market maker that preserves this kind of path independence property is impossible. even if some trolls successfully create a market panic that drops prices to near-zero, when the panic subsides, and prices return to their original levels, the market maker's position will be unchanged even if both the price, and the market maker's balances, made a bunch of crazy moves in the meantime. now, this does not mean that market makers cannot lose money, compared to other holding strategies. if, when you start off, 1 mkr = 5 eth, and then the mkr price moves, and we compare the performance of holding 5 mkr and 12.5 eth in the market maker versus the performance of just holding the assets, the result looks like this: holding a balanced portfolio always wins, except in the case where prices stay exactly the same, in which case the returns of the market maker and the balanced portfolio are equal. hence, the purpose of a market maker of this type is to subsidize guaranteed liquidity as a public good for users, serving as trader of last resort, and not to earn revenue. however, we certainly can modify the market maker to earn revenue, and quite simply: we have it charge a spread. that is, the market maker might charge \(1.005\cdot p\) for buys and offer only \(0.995\cdot p\) for sells. now, being the beneficiary of a market maker becomes a bet: if, in the long run, prices tend to move in one direction, then the market maker loses, at least relative to what they could have gained if they had a balanced portfolio. if, on the other hand, prices tend to bounce around wildly but ultimately come back to the same point, then the market maker can earn a nice profit. this sacrifices the "path independence" property, but in such a way that any deviations from path independence are always in the market maker's favor. there are many designs that path-independent market makers could take; if you are willing to create a token that can issue an unlimited quantity of units, then the "constant reserve ratio" mechanism (where for some constant ratio \(0 \leq r \leq 1\), the token supply is \(p^{1/r 1}\) and the reserve size is \(r \cdot p^{1/r}\) also counts as one, provided that it is implemented correctly and path independence is not compromised by bounds and rounding errors. if you want to make a market maker for existing tokens without a price cap, my favorite (credit to martin koppelmann) mechanism is that which maintains the invariant \(tokena\_balance(p) \cdot tokenb\_balance(p) = k\) for some constant \(k\). so the formulas would be: \(tokena\_balance(p) = \sqrt{k\cdot p}\) \(tokenb\_balance(p) = \sqrt{k/p}\) where \(p\) is the price of \(tokenb\) denominated in \(tokena\). in general, you can make a path-independent market maker by defining any (monotonic) relation between \(tokena\_balance\) and \(tokenb\_balance\) and calculating its derivative at any point to give the price. the above only discusses the role of path independence in preventing one particular type of issue: that where an attacker somehow makes a series of transactions in the context of a series of price movements in order to repeatedly drain the market maker of money. with a path independent market maker, such "money pump" vulnerabilities are impossible. however, there certainly are other kinds of inefficiencies that may exist. if the price of mkr drops from 5 eth to 1 eth, then the market maker used in the example above will have lost 28 eth worth of value, whereas a balanced portfolio would only have lost 20 eth. where did that 8 eth go? in the best case, the price (that is to say, the "real" price, the price level where supply and demand among all users and traders matches up) drops quickly, and some lucky trader snaps up the deal, claiming an 8 eth profit minus negligible transaction fees. but what if there are multiple traders? then, if the price between block \(n\) and block \(n+1\) differs, the fact that traders can bid against each other by setting transaction fees creates an all-pay auction, with revenues going to the miner. as a consequence of the revenue equivalence theorem, we can deduce that we can expect that the transaction fees that traders send into this mechanism will keep going up until they are roughly equal to the size of the profit earned (at least initially; the real equilibrium is for miners to just snap up the money themselves). hence, either way schemes like this are ultimately a gift to the miners. one way to increase social welfare in such a design is to make it possible to create purchase transactions that are only worthwhile for miners to include if they actually make the purchase. that is, if the "real" price of mkr falls from 5 to 4.9, and there are 50 traders racing to arbitrage the market maker, and only the first one of those 50 will make the trade, then only that one should pay the miner a transaction fee. this way, the other 49 failed trades will not clog up the blockchain. eip 86, slated for metropolis, opens up a path toward standardizing such a conditional transaction fee mechanism (another good side effect is that this can also make token sales more unobtrusive, as similar all-pay-auction mechanics apply in many token sales). additionally, there are other inefficiencies if the market maker is the only available trading venue for tokens. for example, if two traders want to exchange a large amount, then they would need to do so via a long series of small buy and sell transactions, needlessly clogging up the blockchain. to mitigate such efficiencies, an on-chain market maker should only be one of the trading venues available, and not the only one. however, this is arguably not a large concern for protocol developers; if there ends up being a demand for a venue for facilitating large-scale trades, then someone else will likely provide it. furthermore, the arguments here only talk about path independence of the market maker assuming a given starting price and ending price. however, because of various psychological effects, as well as multi-equilibrium effects, the ending price is plausibly a function not just of the starting price and recent events that affect the "fundamental" value of the asset, but also of the pattern of trades that happens in response to those events. if a price-dropping event takes place, and because of poor liquidity the price of the asset drops quickly, it may end up recovering to a lower point than if more liquidity had been present in the first place. that said, this may actually be an argument in favor of subsidied market makers: if such multiplier effects exist, then they will have a positive impact on price stability that goes beyond the first-order effect of the liquidity that the market maker itself provides. there is likely a lot of research to be done in determining exactly which path-independent market maker is optimal. there is also the possibility of hybrid semi-automated market makers that have the same guaranteed-liquidity properties, but which include some element of asynchrony, as well as the ability for the operator to "cut in line" and collect the profits in cases where large amounts of capital would otherwise be lost to miners. there is also not yet a coherent theory of just how much (if any) on-chain automated guaranteed liquidity is optimal for various objectives, and to what extent, and by whom, these market makers should be subsidized. all in all, the on-chain mechanism design space is still in its early days, and it's certainly worth much more broadly researching and exploring various options. dark mode toggle sidechains vs plasma vs sharding 2019 jun 12 see all posts special thanks to jinglan wang for review and feedback one question that often comes up is: how exactly is sharding different from sidechains or plasma? all three architectures seem to involve a hub-and-spoke architecture with a central "main chain" that serves as the consensus backbone of the system, and a set of "child" chains containing actual user-level transactions. hashes from the child chains are usually periodically published into the main chain (sharded chains with no hub are theoretically possible but haven't been done so far; this article will not focus on them, but the arguments are similar). given this fundamental similarity, why go with one approach over the others? distinguishing sidechains from plasma is simple. plasma chains are sidechains that have a non-custodial property: if there is any error in the plasma chain, then the error can be detected, and users can safely exit the plasma chain and prevent the attacker from doing any lasting damage. the only cost that users suffer is that they must wait for a challenge period and pay some higher transaction fees on the (non-scalable) base chain. regular sidechains do not have this safety property, so they are less secure. however, designing plasma chains is in many cases much harder, and one could argue that for many low-value applications the security is not worth the added complexity. so what about plasma versus sharding? the key technical difference has to do with the notion of tight coupling. tight coupling is a property of sharding, but not a property of sidechains or plasma, that says that the validity of the main chain ("beacon chain" in ethereum 2.0) is inseparable from the validity of the child chains. that is, a child chain block that specifies an invalid main chain block as a dependency is by definition invalid, and more importantly a main chain block that includes an invalid child chain block is by definition invalid. in non-sharded blockchains, this idea that the canonical chain (ie. the chain that everyone accepts as representing the "real" history) is by definition fully available and valid also applies; for example in the case of bitcoin and ethereum one typically says that the canonical chain is the "longest valid chain" (or, more pedantically, the "heaviest valid and available chain"). in sharded blockchains, this idea that the canonical chain is the heaviest valid and available chain by definition also applies, with the validity and availability requirement applying to both the main chain and shard chains. the new challenge that a sharded system has, however, is that users have no way of fully verifying the validity and availability of any given chain directly, because there is too much data. the challenge of engineering sharded chains is to get around this limitation by giving users a maximally trustless and practical indirect means to verify which chains are fully available and valid, so that they can still determine which chain is canonical. in practice, this includes techniques like committees, snarks/starks, fisherman schemes and fraud and data availability proofs. if a chain structure does not have this tight-coupling property, then it is arguably not a layer-1 sharding scheme, but rather a layer-2 system sitting on top of a non-scalable layer-1 chain. plasma is not a tightly-coupled system: an invalid plasma block absolutely can have its header be committed into the main ethereum chain, because the ethereum base layer has no idea that it represents an invalid plasma block, or even that it represents a plasma block at all; all that it sees is a transaction containing a small piece of data. however, the consequences of a single plasma chain failing are localized to within that plasma chain. sharding try really hard to ensure total validity/availability of every part of the system plasma accept local faults but try to limit their consequences however, if you try to analyze the process of how users perform the "indirect validation" procedure to determine if the chain they are looking at is fully valid and available without downloading and executing the whole thing, one can find more similarities with how plasma works. for example, a common technique used to prevent availability issues is fishermen: if a node sees a given piece of a block as unavailable, it can publish a challenge claiming this, creating a time period within which anyone can publish that piece of data. if a block goes unchallenged for long enough, the blocks and all blocks that cite it as a dependency can be reverted. this seems fundamentally similar to plasma, where if a block is unavailable users can publish a message to the main chain to exit their state in response. both techniques eventually buckle under pressure in the same way: if there are too many false challenges in a sharded system, then users cannot keep track of whether or not all of the availability challenges have been answered, and if there are too many availability challenges in a plasma system then the main chain could get overwhelmed as the exits fill up the chain's block size limit. in both cases, it seems like there's a system that has nominally \(o(c^2)\) scalability (where \(c\) is the computing power of one node) but where scalability falls to \(o(c)\) in the event of an attack. however, sharding has more defenses against this. first of all, modern sharded designs use randomly sampled committees, so one cannot easily dominate even one committee enough to produce a fake block unless one has a large portion (perhaps \(>\frac{1}{3}\)) of the entire validator set of the chain. second, there are better strategies to handling data availability than fishermen: data availability proofs. in a scheme using data availability proofs, if a block is unavailable, then clients' data availability checks will fail and clients will see that block as unavailable. if the block is invalid, then even a single fraud proof will convince them of this fact for an entire block. an \(o(1)\)-sized fraud proof can convince a client of the invalidity of an \(o(c)\)-sized block, and so \(o(c)\) data suffices to convince a client of the invalidity of \(o(c^2)\) data (this is in the worst case where the client is dealing with \(n\) sister blocks all with the same parent of which only one is valid; in more likely cases, one single fraud proof suffices to prove invalidity of an entire invalid chain). hence, sharded systems are theoretically less vulnerable to being overwhelmed by denial-of-service attacks than plasma chains. second, sharded chains provide stronger guarantees in the face of large and majority attackers (with more than \(\frac{1}{3}\) or even \(\frac{1}{2}\) of the validator set). a plasma chain can always be successfully attacked by a 51% attack on the main chain that censors exits; a sharded chain cannot. this is because data availability proofs and fraud proofs happen inside the client, rather than inside the chain, so they cannot be censored by 51% attacks. third, the defenses provided by sharded chains are easier to generalize; plasma's model of exits requires state to be separated into discrete pieces each of which is in the interest of any single actor to maintain, whereas sharded chains relying on data availability proofs, fraud proofs, fishermen and random sampling are theoretically universal. so there really is a large difference between validity and availability guarantees that are provided at layer 2, which are limited and more complex as they require explicit reasoning about incentives and which party has an interest in which pieces of state, and guarantees that are provided by a layer 1 system that is committed to fully satisfying them. but plasma chains also have large advantages too. first, they can be iterated and new designs can be implemented more quickly, as each plasma chain can be deployed separately without coordinating the rest of the ecosystem. second, sharding is inherently more fragile, as it attempts to guarantee absolute and total availability and validity of some quantity of data, and this quantity must be set in the protocol; too little, and the system has less scalability than it could have had, too much, and the entire system risks breaking. the maximum safe level of scalability also depends on the number of users of the system, which is an unpredictable variable. plasma chains, on the other hand, allow different users to make different tradeoffs in this regard, and allow users to adjust more flexibly to changes in circumstances. single-operator plasma chains can also be used to offer more privacy than sharded systems, where all data is public. even where privacy is not desired, they are potentially more efficient, because the total data availability requirement of sharded systems requires a large extra level of redundancy as a safety margin. in plasma systems, on the other hand, data requirements for each piece of data can be minimized, to the point where in the long term each individual piece of data may only need to be replicated a few times, rather than a thousand times as is the case in sharded systems. hence, in the long term, a hybrid system where a sharded base layer exists, and plasma chains exist on top of it to provide further scalability, seems like the most likely approach, more able to serve different groups' of users need than sole reliance on one strategy or the other. and it is unfortunately not the case that at a sufficient level of advancement plasma and sharding collapse into the same design; the two are in some key ways irreducibly different (eg. the data availability checks made by clients in sharded systems cannot be moved to the main chain in plasma because these checks only work if they are done subjectively and based on private information). but both scalability solutions (as well as state channels!) have a bright future ahead of them. slasher ghost, and other developments in proof of stake | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search slasher ghost, and other developments in proof of stake posted by vitalik buterin on october 3, 2014 research & development special thanks to vlad zamfir and zack hess for ongoing research and discussions on proof-of-stake algorithms and their own input into slasher-like proposals one of the hardest problems in cryptocurrency development is that of devising effective consensus algorithms. certainly, relatively passable default options exist. at the very least it is possible to rely on a bitcoin-like proof of work algorithm based on either a randomly-generated circuit approach targeted for specialized-hardware resitance, or failing that simple sha3, and our existing ghost optimizations allow for such an algorithm to provide block times of 12 seconds. however, proof of work as a general category has many flaws that call into question its sustainability as an exclusive source of consensus; 51% attacks from altcoin miners, eventual asic dominance and high energy inefficiency are perhaps the most prominent. over the last few months we have become more and more convinced that some inclusion of proof of stake is a necessary component for long-term sustainability; however, actually implementing a proof of stake algorithm that is effective is proving to be surprisingly complex. the fact that ethereum includes a turing-complete contracting system complicates things further, as it makes certain kinds of collusion much easier without requiring trust, and creates a large pool of stake in the hands of decentralized entities that have the incentive to vote with the stake to collect rewards, but which are too stupid to tell good blockchains from bad. what the rest of this article will show is a set of strategies that deal with most of the issues surrounding proof of stake algorithms as they exist today, and a sketch of how to extend our current preferred proof-of-stake algorithm, slasher, into something much more robust. historical overview: proof of stake and slasher if you're not yet well-versed in the nuances of proof of stake algorithms, first read: https://blog.ethereum.org/2014/07/05/stake/ the fundamental problem that consensus protocols try to solve is that of creating a mechanism for growing a blockchain over time in a decentralized way that cannot easily be subverted by attackers. if a blockchain does not use a consensus protocol to regulate block creation, and simply allows anyone to add a block at any time, then an attacker or botnet with very many ip addresses could flood the network with blocks, and particularly they can use their power to perform double-spend attacks sending a payment for a product, waiting for the payment to be confirmed in the blockchain, and then starting their own "fork" of the blockchain, substituting the payment that they made earlier with a payment to a different account controlled by themselves, and growing it longer than the original so everyone accepts this new blockchain without the payment as truth. the general solution to this problem involves making a block "hard" to create in some fashion. in the case of proof of work, each block requires computational effort to produce, and in the case of proof of stake it requires ownership of coins in most cases, it's a probabilistic process where block-making privileges are doled out randomly in proportion to coin holdings, and in more exotic "negative block reward" schemes anyone can create a block by spending a certain quantity of funds, and they are compensated via transaction fees. in any of these approaches, each chain has a "score" that roughly reflects the total difficulty of producing the chain, and the highest-scoring chain is taken to represent the "truth" at that particular time. for a detailed overview of some of the finer points of proof of stake, see the above-linked article; for those readers who are already aware of the issues i will start off by presenting a semi-formal specification for slasher: blocks are produced by miners; in order for a block to be valid it must satisfy a proof-of-work condition. however, this condition is relatively weak (eg. we can target the mining reward to something like 0.02x the genesis supply every year) every block has a set of designated signers, which are chosen beforehand (see below). for a block with valid pow to be accepted as part of the chain it must be accompanied by signatures from at least two thirds of its designated signers. when block n is produced, we say that the set of potential signers of block n + 3000 is the set of addresses such that sha3(address + block[n].hash) < block[n].balance(address) * d2 where d2 is a difficulty parameter targeting 15 signers per block (ie. if block n has less than 15 signers it goes down otherwise it goes up). note that the set of potential signers is very computationally intensive to fully enumerate, and we don't try to do so; instead we rely on signers to self-declare. if a potential signer for block n + 3000 wants to become a designated signer for that block, they must send a special transaction accepting this responsibility and that transaction must get included between blocks n + 1 and n + 64. the set of designated signers for block n + 3000 is the set of all individuals that do this. this "signer must confirm" mechanism helps ensure that the majority of signers will actually be online when the time comes to sign. for blocks 0 ... 2999, the set of signers is empty, so proof of work alone suffices to create those blocks. when a designated signer adds their signature to block n + 3000, they are scheduled to receive a reward in block n + 6000. if a signer signs two different blocks at height n + 3000, then if someone detects the double-signing before block n + 6000 they can submit an "evidence" transaction containing the two signatures, destroying the signer's reward and transferring a third of it to the whistleblower. if there is an insufficient number of signers to sign at a particular block height h, a miner can produce a block with height h+1 directly on top of the block with height h-1 by mining at an 8x higher difficulty (to incentivize this, but still make it less attractive than trying to create a normal block, there is a 6x higher reward). skipping over two blocks has higher factors of 16x diff and 12x reward, three blocks 32x and 24x, etc. essentially, by explicitly punishing double-signing, slasher in a lot of ways, although not all, makes proof of stake act like a sort of simulated proof of work. an important incidental benefit of slasher is the non-revert property. in proof of work, sometimes after one node mines one block some other node will immediately mine two blocks, and so some nodes will need to revert back one block upon seeing the longer chain. here, every block requires two thirds of the signers to ratify it, and a signer cannot ratify two blocks at the same height without losing their gains in both chains, so assuming no malfeasance the blockchain will never revert. from the point of view of a decentralized application developer, this is a very desirable property as it means that "time" only moves in one direction, just like in a server-based environment. however, slasher is still vulnerable to one particular class of attack: long-range attacks. instead of trying to start a fork from ten blocks behind the current head, suppose that an attacker tries to start a fork starting from ten thousand blocks behind, or even the genesis block all that matters is that the depth of the fork must be greater than the duration of the reward lockup. at that point, because users' funds are unlocked and they can move them to a new address to escape punishment, users have no disincentive against signing on both chains. in fact, we may even expect to see a black market of people selling their old private keys, culminating with an attacker single-handedly acquiring access to the keys that controlled over 50% of the currency supply at some point in history. one approach to solving the long-range double-signing problem is transactions-as-proof-of-stake, an alternative pos solution that does not have an incentive to double-sign because it's the transactions that vote, and there is no reward for sending a transaction (in fact there's a cost, and the reward is outside the network); however, this does nothing to stop the black key market problem. to properly deal with that issue, we will need to relax a hidden assumption. subjective scoring and trust for all its faults, proof of work does have some elegant economic properties. particularly, because proof of work requires an externally rivalrous resource, something with exists and is consumed outside the blockchain, in order to generate blocks (namely, computational effort), launching a fork against a proof of work chain invariably requires having access to, and spending, a large quantity of economic resources. in the case of proof of stake, on the other hand, the only scarce value involved is value within the chain, and between multiple chains that value is not scarce at all. no matter what algorithm is used, in proof of stake 51% of the owners of the genesis block could eventually come together, collude, and produce a longer (ie. higher-scoring) chain than everyone else. this may seem like a fatal flaw, but in reality it is only a flaw if we implicitly accept an assumption that is made in the case of proof of work: that nodes have no knowledge of history. in a proof-of-work protocol, a new node, having no direct knowledge of past events and seeing nothing but the protocol source code and the set of messages that have already been published, can join the network at any point and determine the score of all possible chains, and from there the block that is at the top of the highest-scoring main chain. with proof of stake, as we described, such a property cannot be achieved, since it's very cheap to acquire historical keys and simulate alternate histories. thus, we will relax our assumptions somewhat: we will say that we are only concerned with maintaining consensus between a static set of nodes that are online at least once every n days, allowing these nodes to use their own knowledge of history to reject obvious long-range forks using some formula, and new nodes or long-dormant nodes will need to specify a "checkpoint" (a hash of a block representing what the rest of the network agrees is a recent state) in order to get back onto the consensus. such an approach is essentially a hybrid between the pure and perhaps harsh trust-no-one logic of bitcoin and the total dependency on socially-driven consensus found in networks like ripple. in ripple's case, users joining the system need to select a set of nodes that they trust (or, more precisely, trust not to collude) and rely on those nodes during every step of the consensus process. in the case of bitcoin, the theory is that no such trust is required and the protocol is completely self-contained; the system works just as well between a thousand isolated cavemen with laptops on a thousand islands as it does in a strongly connected society (in fact, it might work better with island cavemen, since without trust collusion is more difficult). in our hybrid scheme, users need only look to the society outside of the protocol exactly once when they first download a client and find a checkpoint and can enjoy bitcoin-like trust properties starting from that point. in order to determine which trust assumption is the better one to take, we ultimately need to ask a somewhat philosophical question: do we want our consensus protocols to exist as absolute cryptoeconomic constructs completely independent of the outside world, or are we okay with relying heavily on the fact that these systems exist in the context of a wider society? although it is indeed a central tenet of mainstream cryptocurrency philosophy that too much external dependence is dangerous, arguably the level of independence that bitcoin affords us in reality is no greater than that provided by the hybrid model. the argument is simple: even in the case of bitcoin, a user must also take a leap of trust upon joining the network first by trusting that they are joining a protocol that contains assets that other people find valuable (eg. how does a user know that bitcoins are worth $380 each and dogecoins only $0.0004? especially with the different capabilities of asics for different algorithms, hashpower is only a very rough estimate), and second by trusting that they are downloading the correct software package. in both the supposedly "pure" model and the hybrid model there is always a need to look outside the protocol exactly once. thus, on the whole, the gain from accepting the extra trust requirement (namely, environmental friendliness and security against oligopolistic mining pools and asic farms) is arguably worth the cost. additionally, we may note that, unlike ripple consensus, the hybrid model is still compatible with the idea of blockchains "talking" to each each other by containing a minimal "light" implementation of each other's protocols. the reason is that, while the scoring mechanism is not "absolute" from the point of view of a node without history suddenly looking at every block, it is perfectly sufficient from the point of view of an entity that remains online over a long period of time, and a blockchain certainly is such an entity. so far, there have been two major approaches that followed some kind of checkpoint-based trust model: developer-issued checkpoints the client developer issues a new checkpoint with each client upgrade (eg. used in ppcoin) revert limit nodes refuse to accept forks that revert more than n (eg. 3000) blocks (eg. used in tendermint) the first approach has been roundly criticized by the cryptocurrency community for being too centralized. the second, however, also has a flaw: a powerful attacker can not only revert a few thousand blocks, but also potentially split the network permanently. in the n-block revert case, the strategy is as follows. suppose that the network is currently at block 10000, and n = 3000. the attacker starts a secret fork, and grows it by 3001 blocks faster than the main network. when the main network gets to 12999, and some node produces block 13000, the attacker reveals his own fork. some nodes will see the main network's block 13000, and refuse to switch to the attacker's fork, but the nodes that did not yet see that block will be happy to revert from 12999 to 10000 and then accept the attacker's fork. from there, the network is permanently split. fortunately, one can actually construct a third approach that neatly solves this problem, which we will call exponentially subjective scoring. essentially, instead of rejecting forks that go back too far, we simply penalize them on a graduating scale. for every block, a node maintains a score and a "gravity" factor, which acts as a multiplier to the contribution that the block makes to the blockchain's score. the gravity of the genesis block is 1, and normally the gravity of any other block is set to be equal to the gravity of its parent. however, if a node receives a block whose parent already has a chain of n descendants (ie. it's a fork reverting n blocks), that block's gravity is penalized by a factor of 0.99n, and the penalty propagates forever down the chain and stacks multiplicatively with other penalties. that is, a fork which starts 1 block ago will need to grow 1% faster than the main chain in order to overtake it, a fork which starts 100 blocks ago will need to grow 2.718 times as quickly, and a fork which starts 3000 blocks ago will need to grow 12428428189813 times as quickly clearly an impossibility with even trivial proof of work. the algorithm serves to smooth out the role of checkpointing, assigning a small "weak checkpoint" role to each individual block. if an attacker produces a fork that some nodes hear about even three blocks earlier than others, those two chains will need to stay within 3% of each other forever in order for a network split to maintain itself. there are other solutions that could be used aside from, or even alongside ess; a particular set of strategies involves stakeholders voting on a checkpoint every few thousand blocks, requiring every checkpoint produced to reflect a large consensus of the majority of the current stake (the reason the majority of the stake can't vote on every block is, of course, that having that many signatures would bloat the blockchain). slasher ghost the other large complexity in implementing proof of stake for ethereum specifically is the fact that the network includes a turing-complete financial system where accounts can have arbitrary permissions and even permissions that change over time. in a simple currency, proof of stake is relatively easy to accomplish because each unit of currency has an unambiguous owner outside the system, and that owner can be counted on to participate in the stake-voting process by signing a message with the private key that owns the coins. in ethereum, however, things are not quite so simple: if we do our job promoting proper wallet security right, the majority of ether is going to be stored in specialized storage contracts, and with turing-complete code there is no clear way of ascertaining or assigning an "owner". one strategy that we looked at was delegation: requiring every address or contract to assign an address as a delegate to sign for them, and that delegate account would have to be controlled by a private key. however, there is a problem with any such approach. suppose that a majority of the ether in the system is actually stored in application contracts (as opposed to personal storage contracts); this includes deposits in schellingcoins and other stake-based protocols, security deposits in probabilistic enforcement systems, collateral for financial derivatives, funds owned by daos, etc. those contracts do not have an owner even in spirit; in that case, the fear is that the contract will default to a strategy of renting out stake-voting delegations to the highest bidder. because attackers are the only entities willing to bid more than the expected return from the delegation, this will make it very cheap for an attacker to acquire the signing rights to large quantities of stake. the only solution to this within the delegation paradigm is to make it extremely risky to dole out signing privileges to untrusted parties; the simplest approach is to modify slasher to require a large deposit, and slash the deposit as well as the reward in the event of double-signing. however, if we do this then we are essentially back to entrusting the fate of a large quantity of funds to a single private key, thereby defeating much of the point of ethereum in the first place. fortunately, there is one alternative to delegation that is somewhat more effective: letting contracts themselves sign. to see how this works, consider the following protocol: there is now a sign opcode added. a signature is a series of virtual transactions which, when sequentially applied to the state at the end of the parent block, results in the sign opcode being called. the nonce of the first vtx in the signature must be the prevhash being signed, the nonce of the second must be the prevhash plus one, and so forth (alternatively, we can make the nonces -1, -2, -3 etc. and require the prevhash to be passed in through transaction data so as to be eventually supplied as an input to the sign opcode). when the block is processed, the state transitions from the vtxs are reverted (this is what is meant by "virtual") but a deposit is subtracted from each signing contract and the contract is registered to receive the deposit and reward in 3000 blocks. basically, it is the contract's job to determine the access policy for signing, and the contract does this by placing the sign opcode behind the appropriate set of conditional clauses. a signature now becomes a set of transactions which together satisfy this access policy. the incentive for contract developers to keep this policy secure, and not dole it out to anyone who asks, is that if it is not secure then someone can double-sign with it and destroy the signing deposit, taking a portion for themselves as per the slasher protocol. some contracts will still delegate, but this is unavoidable; even in proof-of-stake systems for plain currencies such as nxt, many users end up delegating (eg. dpos even goes so far as to institutionalize delegation), and at least here contracts have an incentive to delegate to an access policy that is not likely to come under the influence of a hostile entity in fact, we may even see an equilibrium where contracts compete to deliver secure blockchain-based stake pools that are least likely to double-vote, thereby increasing security over time. however, the virtual-transactions-as-signatures paradigm does impose one complication: it is no longer trivial to provide an evidence transaction showing two signatures by the same signer at the same block height. because the result of a transaction execution depends on the starting state, in order to ascertain whether a given evidence transaction is valid one must prove everything up to the block in which the second signature was given. thus, one must essentially "include" the fork of a blockchain inside of the main chain. to do this efficiently, a relatively simple proposal is a sort of "slasher ghost" protocol, where one can include side-blocks in the main chain as uncles. specifically, we declare two new transaction types: [block_number, uncle_hash] this transaction is valid if (1) the block with the given uncle_hash has already been validated, (2) the block with the given uncle_hash has the given block number, and (3) the parent of that uncle is either in the main chain or was included earlier as an uncle. during the act of processing this transaction, if addresses that double-signed at that height are detected, they are appropriately penalized. [block_number, uncle_parent_hash, vtx] this transaction is valid if (1) the block with the given uncle_parent_hash has already been validated, (2) the given virtual transaction is valid at the given block height with the state at the end of uncle_parent_hash, and (3) the virtual transaction shows a signature by an address which also signed a block at the given block_number in the main chain. this transaction penalizes that one address. essentially, one can think of the mechanism as working like a "zipper", with one block from the fork chain at a time being zipped into the main chain. note that for a fork to start, there must exist double-signers at every block; there is no situation where there is a double-signer 1500 blocks into a fork so a whistleblower must "zip" 1499 innocent blocks into a chain before getting to the target block rather, in such a case, even if 1500 blocks need to be added, each one of them notifies the main chain about five separate malfeasors that double-signed at that height. one somewhat complicated property of the scheme is that the validity of these "slasher uncles" depends on whether or not the node has validated a particular block outside of the main chain; to facilitate this, we specify that a response to a "getblock" message in the wire protocol must include the uncle-dependencies for a block before the actual block. note that this may sometimes lead to a recursive expansion; however, the denial-of-service potential is limited since each individual block still requires a substantial quantity of proof-of-work to produce. blockmakers and overrides finally, there is a third complication. in the hybrid-proof-of-stake version of slasher, if a miner has an overwhelming share of the hashpower, then the miner can produce multiple versions of each block, and send different versions to different parts of the network. half the signers will see and sign one block, half will see and sign another block, and the network will be stuck with two blocks with insufficient signatures, and no signer willing to slash themselves to complete the process; thus, a proof-of-work override will be required, a dangerous situation since the miner controls most of the proof-of-work. there are two possible solutions here: signers should wait a few seconds after receiving a block before signing, and only sign stochastically in some fashion that ensures that a random one of the blocks will dominate. there should be a single "blockmaker" among the signers whose signature is required for a block to be valid. effectively, this transfers the "leadership" role from a miner to a stakeholder, eliminating the problem, but at the cost of adding a dependency on a single party that now has the ability to substantially inconvenience everyone by not signing, or unintentionally by being the target of a denial-of-service attack. such behavior can be disincentivized by having the signer lose part of their deposit if they do not sign, but even still this will result in a rather jumpy block time if the only way to get around an absent blockmaker is using a proof-of-work override. one possible solution to the problem in (2) is to remove proof of work entirely (or almost entirely, keeping a minimal amount for anti-ddos value), replacing it with a mechanism that vlad zamfir has coined "delegated timestamping". essentially, every block must appear on schedule (eg. at 15 second intervals), and when a block appears the signers vote 1 if the block was on time, or 0 if the block was too early or too late. if the majority of the signers votes 0, then the block is treated as invalid kept in the chain in order to give the signers their fair reward, but the blockmaker gets no reward and the state transition gets skipped over. voting is incentivized via schellingcoin the signers whose vote agrees with the majority get an extra reward, so assuming that everyone else is going to be honest everyone has the incentive to be honest, in a self-reinforcing equilibrium. the theory is that a 15-second block time is too fast for signers to coordinate on a false vote (the astute reader may note that the signers were decided 3000 blocks in advance so this is not really true; to fix this we can create two groups of signers, one pre-chosen group for validation and another group chosen at block creation time for timestamp voting). putting it all together taken together, we can thus see something like the following working as a functional version of slasher: every block has a designated blockmaker, a set of designated signers, and a set of designated timestampers. for a block to be accepted as part of the chain it must be accompanied by virtual-transactions-as-signatures from the blockmaker, two thirds of the signers and 10 timestampers, and the block must have some minimal proof of work for anti-ddos reasons (say, targeted to 0.01x per year) during block n, we say that the set of potential signers of block n + 3000 is the set of addresses such that sha3(address + block[n].hash) < block[n].balance(address) * d2 where d2 is a difficulty parameter targeting 15 signers per block (ie. if block n has less than 15 signers it goes down otherwise it goes up). if a potential signer for block n + 3000 wants to become a signer, they must send a special transaction accepting this responsibility and supplying a deposit, and that transaction must get included between blocks n + 1 and n + 64. the set of designated signers for block n + 3000 is the set of all individuals that do this, and the blockmaker is the designated signer with the lowest value for sha3(address + block[n].hash). if the signer set is empty, no block at that height can be made. for blocks 0 ... 2999, the blockmaker and only signer is the protocol developer. the set of timestampers of the block n + 3000 is the set of addresses such that sha3(address + block[n].hash) < block[n].balance(address) * d3, where d3 is targeted such that there is an average of 20 timestampers each block (ie. if block n has less than 20 timestampers it goes down otherwise it goes up). let t be the timestamp of the genesis block. when block n + 3000 is released, timestampers can supply virtual-transactions-as-signatures for that block, and have the choice of voting 0 or 1 on the block. voting 1 means that they saw the block within 7.5 seconds of time t + (n + 3000) * 15, and voting 0 means that they received the block when the time was outside that range. note that nodes should detect if their clocks are out of sync with everyone else's clocks on the blockchain, and if so adjust their system clocks. timestampers who voted along with the majority receive a reward, other timestampers get nothing. the designated signers for block n + 3000 have the ability to sign that block by supplying a set of virtual-transactions-as-a-signature. all designated signers who sign are scheduled to receive a reward and their returned deposit in block n + 6000. signers who skipped out are scheduled to receive their returned deposit minus twice the reward (this means that it's only economically profitable to sign up as a signer if you actually think there is a chance greater than 2/3 that you will be online). if the majority timestamper vote is 1, the blockmaker is scheduled to receive a reward and their returned deposit in block n + 6000. if the majority timestamper vote is 0, the blockmaker is scheduled to receive their deposit minus twice the reward, and the block is ignored (ie. the block is in the chain, but it does not contribute to the chain's score, and the state of the next block starts from the end state of the block before the rejected block). if a signer signs two different blocks at height n + 3000, then if someone detects the double-signing before block n + 6000 they can submit an "evidence" transaction containing the two signatures to either or both chains, destroying the signer's reward and deposit and transferring a third of it to the whistleblower. if there is an insufficient number of signers to sign or the blockmaker is missing at a particular block height h, the designated blockmaker for height h + 1 can produce a block directly on top of the block at height h 1 after waiting for 30 seconds instead of 15. after years of research, one thing has become clear: proof of stake is non-trivial so non-trivial that some even consider it impossible. the issues of nothing-at-stake and long-range attacks, and the lack of mining as a rate-limiting device, require a number of compensatory mechanisms, and even the protocol above does not address the issue of how to randomly select signers. with a substantial proof of work reward, the problem is limited, as block hashes can be a source of randomness and we can mathematically show that the gain from holding back block hashes until a miner finds a hash that favorably selects future signers is usually less than the gain from publishing the block hashes. without such a reward, however, other sources of randomness such as low-influence functions need to be used. for ethereum 1.0, we consider it highly desirable to both not excessively delay the release and not try too many untested features at once; hence, we will likely stick with asic-resistant proof of work, perhaps with non-slasher proof of activity as an addon, and look at moving to a more comprehensive proof of stake model over time. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle quadratic payments: a primer 2019 dec 07 see all posts special thanks to karl floersch and jinglan wang for feedback if you follow applied mechanism design or decentralized governance at all, you may have recently heard one of a few buzzwords: quadratic voting, quadratic funding and quadratic attention purchase. these ideas have been gaining popularity rapidly over the last few years, and small-scale tests have already been deployed: the taiwanese presidential hackathon used quadratic voting to vote on winning projects, gitcoin grants used quadratic funding to fund public goods in the ethereum ecosystem, and the colorado democratic party also experimented with quadratic voting to determine their party platform. to the proponents of these voting schemes, this is not just another slight improvement to what exists. rather, it's an initial foray into a fundamentally new class of social technology which, has the potential to overturn how we make many public decisions, large and small. the ultimate effect of these schemes rolled out in their full form could be as deeply transformative as the industrial-era advent of mostly-free markets and constitutional democracy. but now, you may be thinking: "these are large promises. what do these new governance technologies have that justifies such claims?" private goods, private markets... to understand what is going on, let us first consider an existing social technology: money, and property rights the invisible social technology that generally hides behind money. money and private property are extremely powerful social technologies, for all the reasons classical economists have been stating for over a hundred years. if bob is producing apples, and alice wants to buy apples, we can economically model the interaction between the two, and the results seem to make sense: alice keeps buying apples until the marginal value of the next apple to her is less than the cost of producing it, which is pretty much exactly the optimal thing that could happen. this is all formalized in results such as the "fundamental theorems of welfare economics". now, those of you who have learned some economics may be screaming, but what about imperfect competition? asymmetric information? economic inequality? public goods? externalities? many activities in the real world, including those that are key to the progress of human civilization, benefit (or harm) many people in complicated ways. these activities and the consequences that arise from them often cannot be neatly decomposed into sequences of distinct trades between two parties. but since when do we expect a single package of technologies to solve every problem anyway? "what about oceans?" isn't an argument against cars, it's an argument against car maximalism, the position that we need cars and nothing else. much like how private property and markets deal with private goods, can we try to use economic means to deduce what kind of social technologies would work well for encouraging production of the public goods that we need? ... public goods, public markets private goods (eg. apples) and public goods (eg. public parks, air quality, scientific research, this article...) are different in some key ways. when we are talking about private goods, production for multiple people (eg. the same farmer makes apples for both alice and bob) can be decomposed into (i) the farmer making some apples for alice, and (ii) the farmer making some other apples for bob. if alice wants apples but bob does not, then the farmer makes alice's apples, collects payment from alice, and leaves bob alone. even complex collaborations (the "i, pencil" essay popular in libertarian circles comes to mind) can be decomposed into a series of such interactions. when we are talking about public goods, however, this kind of decomposition is not possible. when i write this blog article, it can be read by both alice and bob (and everyone else). i could put it behind a paywall, but if it's popular enough it will inevitably get mirrored on third-party sites, and paywalls are in any case annoying and not very effective. furthermore, making an article available to ten people is not ten times cheaper than making the article available to a hundred people; rather, the cost is exactly the same. so i either produce the article for everyone, or i do not produce it for anyone at all. so here comes the challenge: how do we aggregate together people's preferences? some private and public goods are worth producing, others are not. in the case of private goods, the question is easy, because we can just decompose it into a series of decisions for each individual. whatever amount each person is willing to pay for, that much gets produced for them; the economics is not especially complex. in the case of public goods, however, you cannot "decompose", and so we need to add up people's preferences in a different way. first of all, let's see what happens if we just put up a plain old regular market: i offer to write an article as long as at least $1000 of money gets donated to me (fun fact: i literally did this back in 2011). every dollar donated increases the probability that the goal will be reached and the article will be published; let us call this "marginal probability" p. at a cost of $k, you can increase the probability that the article will be published by k * p (though eventually the gains will decrease as the probability approaches 100%). let's say to you personally, the article being published is worth $v. would you donate? well, donating a dollar increases the probability it will be published by p, and so gives you an expected $p * v of value. if p * v > 1, you donate, and quite a lot, and if p * v < 1 you don't donate at all. phrased less mathematically, either you value the article enough (and/or are rich enough) to pay, and if that's the case it's in your interest to keep paying (and influencing) quite a lot, or you don't value the article enough and you contribute nothing. hence, the only blog articles that get published would be articles where some single person is willing to basically pay for it themselves (in my experiment in 2011, this prediction was experimentally verified: in most rounds, over half of the total contribution came from a single donor). note that this reasoning applies for any kind of mechanism that involves "buying influence" over matters of public concern. this includes paying for public goods, shareholder voting in corporations, public advertising, bribing politicians, and much more. the little guy has too little influence (not quite zero, because in the real world things like altruism exist) and the big guy has too much. if you had an intuition that markets work great for buying apples, but money is corrupting in "the public sphere", this is basically a simplified mathematical model that shows why. we can also consider a different mechanism: one-person-one-vote. let's say you can either vote that i deserve a reward for writing this article, or you can vote that i don't, and my reward is proportional to the number of votes in my favor. we can interpret this as follows: your first "contribution" costs only a small amount of effort, so you'll support an article if you care about it enough, but after that point there is no more room to contribute further; your second contribution "costs" infinity. now, you might notice that neither of the graphs above look quite right. the first graph over-privileges people who care a lot (or are wealthy), the second graph over-privileges people who care only a little, which is also a problem. the single sheep's desire to live is more important than the two wolves' desire to have a tasty dinner. but what do we actually want? ultimately, we want a scheme where how much influence you "buy" is proportional to how much you care. in the mathematical lingo above, we want your k to be proportional to your v. but here's the problem: your v determines how much you're willing to pay for one unit of influence. if alice were willing to pay $100 for the article if she had to fund it herself, then she would be willing to pay $1 for an increased 1% chance it will get written, and if bob were only willing to pay $50 for the article then he would only be willing to pay $0.5 for the same "unit of influence". so how do we match these two up? the answer is clever: your n'th unit of influence costs you $n . that is, for example, you could buy your first vote for $0.01, but then your second would cost $0.02, your third $0.03, and so forth. suppose you were alice in the example above; in such a system she would keep buying units of influence until the cost of the next one got to $1, so she would buy 100 units. bob would similarly buy until the cost got to $0.5, so he would buy 50 units. alice's 2x higher valuation turned into 2x more units of influence purchased. let's draw this as a graph: now let's look at all three beside each other: one dollar one vote quadratic voting one person one vote notice that only quadratic voting has this nice property that the amount of influence you purchase is proportional to how much you care; the other two mechanisms either over-privilege concentrated interests or over-privilege diffuse interests. now, you might ask, where does the quadratic come from? well, the marginal cost of the n'th vote is $n (or $0.01 * n), but the total cost of n votes is \(\approx \frac{n^2}{2}\). you can view this geometrically as follows: the total cost is the area of a triangle, and you probably learned in math class that area is base * height / 2. and since here base and height are proportionate, that basically means that total cost is proportional to number of votes squared hence, "quadratic". but honestly it's easier to think "your n'th unit of influence costs $n". finally, you might notice that above i've been vague about what "one unit of influence" actually means. this is deliberate; it can mean different things in different contexts, and the different "flavors" of quadratic payments reflect these different perspectives. quadratic voting see also the original paper: https://papers.ssrn.com/sol3/papers.cfm?abstract%5fid=2003531 let us begin by exploring the first "flavor" of quadratic payments: quadratic voting. imagine that some organization is trying to choose between two choices for some decision that affects all of its members. for example, this could be a company or a nonprofit deciding which part of town to make a new office in, or a government deciding whether or not to implement some policy, or an internet forum deciding whether or not its rules should allow discussion of cryptocurrency prices. within the context of the organization, the choice made is a public good (or public bad, depending on whom you talk to): everyone "consumes" the results of the same decision, they just have different opinions about how much they like the result. this seems like a perfect target for quadratic voting. the goal is that option a gets chosen if in total people like a more, and option b gets chosen if in total people like b more. with simple voting ("one person one vote"), the distinction between stronger vs weaker preferences gets ignored, so on issues where one side is of very high value to a few people and the other side is of low value to more people, simple voting is likely to give wrong answers. with a private-goods market mechanism where people can buy as many votes as they want at the same price per vote, the individual with the strongest preference (or the wealthiest) carries everything. quadratic voting, where you can make n votes in either direction at a cost of n2, is right in the middle between these two extremes, and creates the perfect balance. note that in the voting case, we're deciding two options, so different people will favor a over b or b over a; hence, unlike the graphs we saw earlier that start from zero, here voting and preference can both be positive or negative (which option is considered positive and which is negative doesn't matter; the math works out the same way) as shown above, because the n'th vote has a cost of n, the number of votes you make is proportional to how much you value one unit of influence over the decision (the value of the decision multiplied by the probability that one vote will tip the result), and hence proportional to how much you care about a being chosen over b or vice versa. hence, we once again have this nice clean "preference adding" effect. we can extend quadratic voting in multiple ways. first, we can allow voting between more than two options. while traditional voting schemes inevitably fall prey to various kinds of "strategic voting" issues because of arrow's theorem and duverger's law, quadratic voting continues to be optimal in contexts with more than two choices. the intuitive argument for those interested: suppose there are established candidates a and b and new candidate c. some people favor c > a > b but others c > b > a. in a regular vote, if both sides think c stands no chance, they decide may as well vote their preference between a and b, so c gets no votes, and c's failure becomes a self-fulfilling prophecy. in quadratic voting the former group would vote [a +10, b -10, c +1] and the latter [a -10, b +10, c +1], so the a and b votes cancel out and c's popularity shines through. second, we can look not just at voting between discrete options, but also at voting on the setting of a thermostat: anyone can push the thermostat up or down by 0.01 degrees n times by paying a cost of n2. plot twist: the side wanting it colder only wins when they convince the other side that "c" stands for "caliente". quadratic funding see also the original paper: https://papers.ssrn.com/sol3/papers.cfm?abstract%5fid=3243656 quadratic voting is optimal when you need to make some fixed number of collective decisions. but one weakness of quadratic voting is that it doesn't come with a built-in mechanism for deciding what goes on the ballot in the first place. proposing votes is potentially a source of considerable power if not handled with care: a malicious actor in control of it can repeatedly propose some decision that a majority weakly approves of and a minority strongly disapproves of, and keep proposing it until the minority runs out of voting tokens (if you do the math you'll see that the minority would burn through tokens much faster than the majority). let's consider a flavor of quadratic payments that does not run into this issue, and makes the choice of decisions itself endogenous (ie. part of the mechanism itself). in this case, the mechanism is specialized for one particular use case: individual provision of public goods. let us consider an example where someone is looking to produce a public good (eg. a developer writing an open source software program), and we want to figure out whether or not this program is worth funding. but instead of just thinking about one single public good, let's create a mechanism where anyone can raise funds for what they claim to be a public good project. anyone can make a contribution to any project; a mechanism keeps track of these contributions and then at the end of some period of time the mechanism calculates a payment to each project. the way that this payment is calculated is as follows: for any given project, take the square root of each contributor's contribution, add these values together, and take the square of the result. or in math speak: \[(\sum_{i=1}^n \sqrt{c_i})^2\] if that sounds complicated, here it is graphically: in any case where there is more than one contributor, the computed payment is greater than the raw sum of contributions; the difference comes out of a central subsidy pool (eg. if ten people each donate $1, then the sum-of-square-roots is $10, and the square of that is $100, so the subsidy is $90). note that if the subsidy pool is not big enough to make the full required payment to every project, we can just divide the subsidies proportionately by whatever constant makes the totals add up to the subsidy pool's budget; you can prove that this solves the tragedy-of-the-commons problem as well as you can with that subsidy budget. there are two ways to intuitively interpret this formula. first, one can look at it through the "fixing market failure" lens, a surgical fix to the tragedy of the commons problem. in any situation where alice contributes to a project and bob also contributes to that same project, alice is making a contribution to something that is valuable not only to herself, but also to bob. when deciding how much to contribute, alice was only taking into account the benefit to herself, not bob, whom she most likely does not even know. the quadratic funding mechanism adds a subsidy to compensate for this effect, determining how much alice "would have" contributed if she also took into account the benefit her contribution brings to bob. furthermore, we can separately calculate the subsidy for each pair of people (nb. if there are n people there are n * (n-1) / 2 pairs), and add up all of these subsidies together, and give bob the combined subsidy from all pairs. and it turns out that this gives exactly the quadratic funding formula. second, one can look at the formula through a quadratic voting lens. we interpret the quadratic funding as being a special case of quadratic voting, where the contributors to a project are voting for that project and there is one imaginary participant voting against it: the subsidy pool. every "project" is a motion to take money from the subsidy pool and give it to that project's creator. everyone sending \(c_i\) of funds is making \(\sqrt{c_i}\) votes, so there's a total of \(\sum_{i=1}^n \sqrt{c_i}\) votes in favor of the motion. to kill the motion, the subsidy pool would need to make more than \(\sum_{i=1}^n \sqrt{c_i}\) votes against it, which would cost it more than \((\sum_{i=1}^n \sqrt{c_i})^2\). hence, \((\sum_{i=1}^n \sqrt{c_i})^2\) is the maximum transfer from the subsidy pool to the project that the subsidy pool would not vote to stop. quadratic funding is starting to be explored as a mechanism for funding public goods already; gitcoin grants for funding public goods in the ethereum ecosystem is currently the biggest example, and the most recent round led to results that, in my own view, did a quite good job of making a fair allocation to support projects that the community deems valuable. numbers in white are raw contribution totals; numbers in green are the extra subsidies. quadratic attention payments see also the original post: https://kortina.nyc/essays/speech-is-free-distribution-is-not-a-tax-on-the-purchase-of-human-attention-and-political-power/ one of the defining features of modern capitalism that people love to hate is ads. our cities have ads: source: https://www.flickr.com/photos/argonavigo/36657795264 our subway turnstiles have ads: source: https://commons.wikimedia.org/wiki/file:nyc,_subway_ad_on_prince_st.jpg our politics are dominated by ads: source: https://upload.wikimedia.org/wikipedia/commons/e/e3/billboard_challenging_the_validity_of_barack_obama%27s_birth_certificate.jpg and even the rivers and the skies have ads. now, there are some places that seem to not have this problem: but really they just have a different kind of ads: now, recently there are attempts to move beyond this in some cities. and on twitter. but let's look at the problem systematically and try to see what's going wrong. the answer is actually surprisingly simple: public advertising is the evil twin of public goods production. in the case of public goods production, there is one actor that is taking on an expenditure to produce some product, and this product benefits a large number of people. because these people cannot effectively coordinate to pay for the public goods by themselves, we get much less public goods than we need, and the ones we do get are those favored by wealthy actors or centralized authorities. here, there is one actor that reaps a large benefit from forcing other people to look at some image, and this action harms a large number of people. because these people cannot effectively coordinate to buy out the slots for the ads, we get ads we don't want to see, that are favored by... wealthy actors or centralized authorities. so how do we solve this dark mirror image of public goods production? with a bright mirror image of quadratic funding: quadratic fees! imagine a billboard where anyone can pay $1 to put up an ad for one minute, but if they want to do this multiple times the prices go up: $2 for the second minute, $3 for the third minute, etc. note that you can pay to extend the lifetime of someone else's ad on the billboard, and this also costs you only $1 for the first minute, even if other people already paid to extend the ad's lifetime many times. we can once again interpret this as being a special case of quadratic voting: it's basically the same as the "voting on a thermostat" example above, but where the thermostat in question is the number of seconds an ad stays up. this kind of payment model could be applied in cities, on websites, at conferences, or in many other contexts, if the goal is to optimize for putting up things that people want to see (or things that people want other people to see, but even here it's much more democratic than simply buying space) rather than things that wealthy people and centralized institutions want people to see. complexities and caveats perhaps the biggest challenge to consider with this concept of quadratic payments is the practical implementation issue of identity and bribery/collusion. quadratic payments in any form require a model of identity where individuals cannot easily get as many identities as they want: if they could, then they could just keep getting new identities and keep paying $1 to influence some decision as many times as they want, and the mechanism collapses into linear vote-buying. note that the identity system does not need to be airtight (in the sense of preventing multiple-identity acquisition), and indeed there are good civil-liberties reasons why identity systems probably should not try to be airtight. rather, it just needs to be robust enough that manipulation is not worth the cost. collusion is also tricky. if we can't prevent people from selling their votes, the mechanisms once again collapse into one-dollar-one-vote. we don't just need votes to be anonymous and private (while still making the final result provable and public); we need votes to be so private that even the person who made the vote can't prove to anyone else what they voted for. this is difficult. secret ballots do this well in the offline world, but secret ballots are a nineteenth century technology, far too inefficient for the sheer amount of quadratic voting and funding that we want to see in the twenty first century. fortunately, there are technological means that can help, combining together zero-knowledge proofs, encryption and other cryptographic technologies to achieve the precise desired set of privacy and verifiability properties. there's also proposed techniques to verify that private keys actually are in an individual's possession and not in some hardware or cryptographic system that can restrict how they use those keys. however, these techniques are all untested and require quite a bit of further work. another challenge is that quadratic payments, being a payment-based mechanism, continues to favor people with more money. note that because the cost of votes is quadratic, this effect is dampened: someone with 100 times more money only has 10 times more influence, not 100 times, so the extent of the problem goes down by 90% (and even more for ultra-wealthy actors). that said, it may be desirable to mitigate this inequality of power further. this could be done either by denominating quadratic payments in a separate token of which everyone gets a fixed number of units, or giving each person an allocation of funds that can only be used for quadratic-payments use cases: this is basically andrew yang's "democracy dollars" proposal. a third challenge is the "rational ignorance" and "rational irrationality" problems, which is that decentralized public decisions have the weakness that any single individual has very little effect on the outcome, and so little motivation to make sure they are supporting the decision that is best for the long term; instead, pressures such as tribal affiliation may dominate. there are many strands of philosophy that emphasize the ability of large crowds to be very wrong despite (or because of!) their size, and quadratic payments in any form do little to address this. quadratic payments do better at mitigating this problem than one-person-one-vote systems, and these problems can be expected to be less severe for medium-scale public goods than for large decisions that affect many millions of people, so it may not be a large challenge at first, but it's certainly an issue worth confronting. one approach is combining quadratic voting with elements of sortition. another, potentially more long-term durable, approach is to combine quadratic voting with another economic technology that is much more specifically targeted toward rewarding the "correct contrarianism" that can dispel mass delusions: prediction markets. a simple example would be a system where quadratic funding is done retrospectively, so people vote on which public goods were valuable some time ago (eg. even 2 years), and projects are funded up-front by selling shares of the results of these deferred votes; by buying shares people would be both funding the projects and betting on which project would be viewed as successful in 2 years' time. there is a large design space to experiment with here. conclusion as i mentioned at the beginning, quadratic payments do not solve every problem. they solve the problem of governing resources that affect large numbers of people, but they do not solve many other kinds of problems. a particularly important one is information asymmetry and low quality of information in general. for this reason, i am a fan of techniques such as prediction markets (see electionbettingodds.com for one example) to solve information-gathering problems, and many applications can be made most effective by combining different mechanisms together. one particular cause dear to me personally is what i call "entrepreneurial public goods": public goods that in the present only a few people believe are important but in the future many more people will value. in the 19th century, contributing to abolition of slavery may have been one example; in the 21st century i can't give examples that will satisfy every reader because it's the nature of these goods that their importance will only become common knowledge later down the road, but i would point to life extension and ai risk research as two possible examples. that said, we don't need to solve every problem today. quadratic payments are an idea that has only become popular in the last few years; we still have not seen more than small-scale trials of quadratic voting and funding, and quadratic attention payments have not been tried at all! there is still a long way to go. but if we can get these mechanisms off the ground, there is a lot that these mechanisms have to offer! state of ethereum: august edition | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search state of ethereum: august edition posted by vitalik buterin on august 27, 2014 research & development development of ethereum has been progressing increasingly quickly this past month. the release of poc5 ("proof of concept five") last month the day before the sale marked an important event for the project, as for the first time we had two clients, one written in c++ and one in go, perfectly interoperating with each other and processing the same blockchain. two weeks later, the python client was also added to the list, and now a java version is also almost done. currently, we are in the process of using an initial quantity of funds that we have already withdrawn from the ethereum exodus address to expand our operations, and we are hard at work implementing poc6, the next version in the series, which features a number of enhancements. at this point, ethereum is at a state roughly similar to bitcoin in mid-2009; the clients and protocol work, and people can send transactions and build decentralized applications with contracts and even pretty user interfaces inside of html and javascript, but the software is inefficient, the ui underdeveloped, networking-level inefficiencies and vulnerabilities will take a while to get rooted out, and there is a very high risk of security holes and consensus failures. in order to be comfortable releasing ethereum 1.0, there are only four things that absolutely need to be done: protocol and network-level security testing, virtual machine efficiency upgrades, a very large battery of tests to ensure inter-client compatibility, and a finalized consensus algorithm. all of these are now high on our priority list; but at the same time we are also working in parallel on powerful and easy-to-use tools for building decentralized applications, contract standard libraries, better user interfaces, light clients, and all of the other small features that push the development experience from good to best. poc6 the major changes that are scheduled for poc6 are as follows: the block time is decreased from 60 seconds to 12 seconds, using a new ghost-based protocol that expands upon our previous efforts at reducing the block time to 60 seconds the addmod and mulmod (unsigned modular addition and unsigned modular multiplication) are added at slots 0x14 and 0x15, respectively. the purpose of these is to make it easier to implement certain kinds of number-theoretic cryptographic algorithms, eg. elliptic curve signature verification. see here for some example code that uses these operations. the opcodes dup and swap are removed from their current slots. instead, we have the new opcodes dup1, dup2 ... dup16 at positions 0x80 ... 0x8f and similarly swap1 ... swap16 at positions 0x90 ... 0x9f. dupn copies the nth highest value in the stack to the top of the stack, and swapn swaps the highest and (n+1)-th highest value on the stack. the with statement is added to serpent, as a manual way of using these opcodes to more efficiently access variables. example usage is found here. note that this is an advanced feature, and has a limitation: if you stack so many layers of nesting beneath a with statement that you end up trying to access a variable more than 16 stack levels deep, compilation will fail. eventually, the hope is that the serpent compiler will intelligently choose between stack-based variables and memory-based variables as needed to maximize efficiency. the post opcode is added at slot 0xf3. post is similar to call, except that (1) the opcode has 5 inputs and 0 outputs (ie. it does not return anything), and (2) the execution happens asynchronously, after everything else is finished. more precisely, the process of transaction execution now involves (1) initializing a "post queue" with the message embedded in the transaction, (2) repeatedly processing the first message in the post queue until the post queue is empty, and (3) refunding gas to the transaction origin and processing suicides. post adds a message to the post queue. the hash of a block is now the hash of the header, and not the entire block (which is how it really should have been all along), the code hash for accounts with no code is "" instead of sha3("") (making all non-contract accounts 32 bytes more efficient), and the to address for contract creation transactions is now the empty string instead of twenty zero bytes. on efficiency aside from these changes, the one major idea that we are beginning to develop is the concept of "native contract extensions". the idea comes from long internal and external discussions about the tradeoffs between having a more reduced instruction set ("risc") in our virtual machine, limited to basic memory, storage and blockchain interaction, sub-calls and arithmetic, and a more complex instruction set ("cisc"), including features such as elliptic curve signature verification, a wider library of hash algorithms, bloom filters, and data structures such as heaps. the argument in favor of the reduced instruction set is twofold. first, it makes the virtual machine simpler, allowing for easier development of multiple implementations and reducing the risk of security issues and consensus failures. second, no specific set of opcodes will ever encompass everything that people will want to do, so a more generalized solution would be much more future-proof. the argument in favor of having more opcodes is simple efficiency. as an example, consider the heap). a heap is a data structure which supports three operations: adding a value to the heap, quickly checking the current smallest value on the heap, and removing the smallest value from the heap. heaps are particularly useful when building decentralized markets; the simplest way to design a market is to have a heap of sell orders, an inverted (ie. highest-first) heap of buy orders, and repeatedly pop the top buy and sell orders off the heap and match them with each other while the ask price is greater than the bid. the way to do this relatively quickly, in logarithmic time for adding and removing and constant time for checking, is using a tree: the key invariant is that the parent node of a tree is always lower than both of its children. the way to add a value to the tree is to add it to the end of the bottom level (or the start of a new bottom level if the current bottom level is full), and then to move the node up the tree, swapping it with its parents, for as long as the parent is higher than the child. at the end of the process, the invariant is again satisfied with the new node being in the tree at the right place: to remove a node, we pop off the node at the top, take a node out from the bottom level and move it into its place, and then move that node down the tree as deep as makes sense: and to see what the lowest node is, we, well, look at the top. the key point here is that both of these operations are logarithmic in the number of nodes in the tree; even if your heap has a billion items, it takes only 30 steps to add or remove a node. it's a nontrivial exercise in computer science, but if you're used to dealing with trees it's not particularly complicated. now, let's try to implement this in ethereum code. the full code sample for this is here; for those interested the parent directory also contains a batched market implementation using these heaps and an attempt at implementing futarchy using the markets. here is a code sample for the part of the heap algorithm that handles adding new values: # push if msg.data[0] == 0: sz = contract.storage[0] contract.storage[sz + 1] = msg.data[1] k = sz + 1 while k > 1: bottom = contract.storage[k] top = contract.storage[k/2] if bottom < top: contract.storage[k] = top contract.storage[k/2] = bottom k /= 2 else: k = 0 contract.storage[0] = sz + 1 the model that we use is that contract.storage[0] stores the size (ie. number of values) of the heap, contract.storage[1] is the root node, and from there for any n <= contract.storage[0], contract.storage[n] is a node with parent contract.storage[n/2] and children contract.storage[n*2] and contract.storage[n*2+1] (if n*2 and n*2+1 are less than or equal to the heap size, of course). relatively simple. now, what's the problem? in short, as we already mentioned, the primary concern is inefficiency. theoretically, all tree-based algorithms have most of their operations take log(n) time. here, however, the problem is that what we actually have is a tree (the heap) on top of a tree (the ethereum patricia tree storing the state) on top of a tree (leveldb). hence, the market designed here actually has log3(n) overhead in practice, a rather substantial slowdown. as another example, over the last several days i have written, profiled and tested serpent code for elliptic curve signature verification. the code is basically a fairly simple port of pybitcointools, albeit some uses of recursion have been replaced with loops in order to increase efficiency. even still, the gas cost is staggering: an average of about 340000 for one signature verification. and this, mind you, is after adding some optimizations. for example, see the code for taking modular exponents: with b = msg.data[0]: with e = msg.data[1]: with m = msg.data[2]: with o = 1: with bit = 2 ^ 255: while gt(bit, 0): # a touch of loop unrolling for 20% efficiency gain o = mulmod(mulmod(o, o, m), b ^ !(!(e & bit)), m) o = mulmod(mulmod(o, o, m), b ^ !(!(e & div(bit, 2))), m) o = mulmod(mulmod(o, o, m), b ^ !(!(e & div(bit, 4))), m) o = mulmod(mulmod(o, o, m), b ^ !(!(e & div(bit, 8))), m) bit = div(bit, 16) return(o) this takes up 5084 gas for any input. it is still a fairly simple algorithm; a more advanced implementation may be able to speed this up by up to 50%, but even still iterating over 256 bits is expensive no matter what you do. what these two examples show is that high-performance, high-volume decentralized applications are in some cases going to be quite difficult to write on top of ethereum without either complex instructions to implement heaps, signature verification, etc in the protocol, or something to replace them. the mechanism that we are now working on is an attempt conceived by our lead developer gavin wood to essentially get the best of both worlds, preserving the generality of simple instructions but at the same time getting the speed of natively implemented operations: native code extensions. native code extensions the way that native code extensions work is as follows. suppose that there exists some operation or data structure that we want ethereum contracts to have access to, but which we can optimize by writing an implementation in c++ or machine code. what we do is we first write an implementation in ethereum virtual machine code, test it and make sure it works, and publish that implementation as a contract. we then either write or find an implementation that handles this task natively, and add a line of code to the message execution engine which looks for calls to the contract that we created, and instead of sub-calling the virtual machine calls the native extension instead. hence, instead of it taking 22 seconds to run the elliptic curve recovery operation, it would take only 0.02 seconds. the problem is, how do we make sure that the fees on these native extensions are not prohibitive? this is where it gets tricky. first, let's make a few simplifications, and see where the economic analysis leads. suppose that miners have access to a magic oracle that tells them the maximum amount of time that a given contract can take. without native extensions, this magic oracle exists now it consists simply of looking at the startgas of the transaction but it becomes not quite so simple when you have a contract whose startgas is 1000000 and which looks like it may or may not call a few native extensions to speed things up drastically. but suppose that it exists. now, suppose that a user comes in with a transaction spending 1500 gas on miscellaneous business logic and 340000 gas on an optimized elliptic curve operation, which actually costs only the equivalent of 500 gas of normal execution to compute. suppose that the standard market-rate transaction fee is 1 szabo (ie. micro-ether) per gas. the user sets a gasprice of 0.01 szabo, effectively paying for 3415 gas, because he would be unwilling to pay for the entire 341500 gas for the transaction but he knows that miners can process his transaction for 2000 gas' worth of effort. the user sends the transaction, and a miner receives it. now, there are going to be two cases: the miner has enough unconfirmed transactions in its mempool and is willing to expend the processing power to produce a block where the total gas used brushes against the block-level gas limit (this, to remind you, is 1.2 times the long-term exponential moving average of the gas used in recent blocks). in this case, the miner has a static amount of gas to fill up, so it wants the highest gasprice it can get, so the transaction paying 0.01 szabo per gas instead of the market rate of 1 szabo per gas gets unceremoniously discarded. either not enough unconfirmed transactions exist, or the miner is small and not willing or able to process every transaction. in this case, the dominating factor in whether or not a transaction is accepted is the ratio of reward to processing time. hence, the miner's incentives are perfectly aligned, and since this transaction has a 70% better reward to cost rate than most others it will be accepted. what we see is that, given our magic oracle, such transactions will be accepted, but they will take a couple of extra blocks to get into the network. over time, the block-level gas limit would rise as more contract extensions are used, allowing the use of even more of them. the primary worry is that if such mechanisms become too prevalent, and the average block's gas consumption would be more than 99% native extensions, then the regulatory mechanism preventing large miners from creating extremely large blocks as a denial-of-service attack on the network would be weakened at a gas limit of 1000000000, a malicious miner could make an unoptimized contract that takes up that many computational steps, and freeze the network. so altogether we have two problems. one is the theoretical problem of the gaslimit becoming a weaker safeguard, and the other is the fact that we don't have a magic oracle. fortunately, we can solve the second problem, and in doing so at the same time limit the effect of the first problem. the naive solution is simple: instead of gasprice being just one value, there would be one default gasprice and then a list of [address, gasprice] pairs for specific contracts. as soon as execution enters an eligible contract, the virtual machine would keep track of how much gas it used within that scope, and then appropriately refund the transaction sender at the end. to prevent gas counts from getting too out of hand, the secondary gas prices would be required to be at least 1% (or some other fraction) of the original gasprice. the problem is that this mechanism is space-inefficient, taking up about 25 extra bytes per contract. a possible fix is to allow people to register tables on the blockchain, and then simply refer to which fee table they wish to use. in any case, the exact mechanism is not finalized; hence, native extensions may end up waiting until poc7. mining the other change that will likely begin to be introduced in poc7 is a new mining algorithm. we (well, primarily vlad zamfir) have been slowly working on the mining algorithm in our mining repo, to the point where there is a working proof of concept, albeit more research is required to continue to improve its asic resistance. the basic idea behind the algorithm is essentially to randomly generate a new circuit every 1000 nonces; a device capable of processing this algorithm would need to be capable of processing all circuits that could be generated, and theoretically there should exist some circuit that conceivably could be generated by our system that would be equivalent to sha256, or blake, or keccak, or any other algorithms in x11. hence, such a device would have to be a generalized computer essentially, the aim is something that tried to approach mathematically provable specialization-resistance. in order to make sure that all hash functions generated are secure, a sha3 is always applied at the end. of course, perfect specialization-resistance is impossible; there will always be some features of a cpu that will prove to be extraneous in such an algorithm, so a nonzero theoretical asic speedup is inevitable. currently, the biggest threat to our approach is likely some kind of rapidly switching fpga. however, there is an economic argument which shows that cpus will survive even if asics have a speedup, as long as that speedup is low enough; see my earlier article on mining for an overview of some of the details. a possible tradeoff that we will have to make is whether or not to make the algorithm memory-hard; asic resistance is hard enough as it stands, and memory-hardness may or may not end up interfering with that goal (cf. peter todd's arguments that memory-based algorithms may actually encourage centralization); if the algorithm is not memory-hard, then it may end up being gpu-friendly. at the same time, we are looking into hybrid-proof-of-stake scoring functions as a way of augmenting pow with further security, requiring 51% attacks to simultaneously have a large economic component. with the protocol in an increasingly stable state, another area in which it is time to start developing is what we are starting to call "ethereum 1.5" mechanisms on top of ethereum as it stands today, without the need for any new adjustments to the core protocol, that allow for increased scalability and efficiency for contracts and decentralized applications, either by cleverly combining and batching transactions or by using the blockchain only as a backup enforcement mechanism with only the nodes that care about a particular contract running that contract by default. there are a number of mechanism in this category; this is something that will see considerably increased attention from both ourselves and hopefully others in the community. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle [mirror] exploring elliptic curve pairings 2017 jan 14 see all posts this is a mirror of the post at https://medium.com/@vitalikbuterin/exploring-elliptic-curve-pairings-c73c1864e627 trigger warning: math. one of the key cryptographic primitives behind various constructions, including deterministic threshold signatures, zk-snarks and other simpler forms of zero-knowledge proofs is the elliptic curve pairing. elliptic curve pairings (or "bilinear maps") are a recent addition to a 30-year-long history of using elliptic curves for cryptographic applications including encryption and digital signatures; pairings introduce a form of "encrypted multiplication", greatly expanding what elliptic curve-based protocols can do. the purpose of this article will be to go into elliptic curve pairings in detail, and explain a general outline of how they work. you're not expected to understand everything here the first time you read it, or even the tenth time; this stuff is genuinely hard. but hopefully this article will give you at least a bit of an idea as to what is going on under the hood. elliptic curves themselves are very much a nontrivial topic to understand, and this article will generally assume that you know how they work; if you do not, i recommend this article here as a primer: https://blog.cloudflare.com/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/. as a quick summary, elliptic curve cryptography involves mathematical objects called "points" (these are literal two-dimensional points with \((x, y)\) coordinates), with special formulas for adding and subtracting them (ie. for calculating the coordinates of \(r = p + q\)), and you can also multiply a point by an integer (ie. \(p \cdot n = p + p + ... + p\), though there's a much faster way to compute it if \(n\) is big). here's how point addition looks like graphically. there exists a special point called the "point at infinity" (\(o\)), the equivalent of zero in point arithmetic; it's always the case that \(p + o = p\). also, a curve has an "order"; there exists a number \(n\) such that \(p \cdot n = o\) for any \(p\) (and of course, \(p \cdot (n+1) = p, p \cdot (7 \cdot n + 5) = p \cdot 5\), and so on). there is also some commonly agreed upon "generator point" \(g\), which is understood to in some sense represent the number \(1\). theoretically, any point on a curve (except \(o\)) can be \(g\); all that matters is that \(g\) is standardized. pairings go a step further in that they allow you to check certain kinds of more complicated equations on elliptic curve points — for example, if \(p = g \cdot p, q = g \cdot q\) and \(r = g \cdot r\), you can check whether or not \(p \cdot q = r\), having just \(p, q\) and \(r\) as inputs. this might seem like the fundamental security guarantees of elliptic curves are being broken, as information about \(p\) is leaking from just knowing p, but it turns out that the leakage is highly contained — specifically, the decisional diffie hellman problem is easy, but the computational diffie hellman problem (knowing \(p\) and \(q\) in the above example, computing \(r = g \cdot p \cdot q\)) and the discrete logarithm problem (recovering \(p\) from \(p\)) remain computationally infeasible (at least, if they were before). a third way to look at what pairings do, and one that is perhaps most illuminating for most of the use cases that we are about, is that if you view elliptic curve points as one-way encrypted numbers (that is, \(encrypt(p) = p \cdot g = p\)), then whereas traditional elliptic curve math lets you check linear constraints on the numbers (eg. if \(p = g \cdot p, q = g \cdot q\) and \(r = g \cdot r\), checking \(5 \cdot p + 7 \cdot q = 11 \cdot r\) is really checking that \(5 \cdot p + 7 \cdot q = 11 \cdot r\)), pairings let you check quadratic constraints (eg. checking \(e(p, q) \cdot e(g, g \cdot 5) = 1\) is really checking that \(p \cdot q + 5 = 0\)). and going up to quadratic is enough to let us work with deterministic threshold signatures, quadratic arithmetic programs and all that other good stuff. now, what is this funny \(e(p, q)\) operator that we introduced above? this is the pairing. mathematicians also sometimes call it a bilinear map; the word "bilinear" here basically means that it satisfies the constraints: \(e(p, q + r) = e(p, q) \cdot e(p, r)\) \(e(p + s, q) = e(p, q) \cdot e(s, q)\) note that \(+\) and \(\cdot\) can be arbitrary operators; when you're creating fancy new kinds of mathematical objects, abstract algebra doesn't care how \(+\) and \(\cdot\) are defined, as long as they are consistent in the usual ways, eg. \(a + b = b + a, (a \cdot b) \cdot c = a \cdot (b \cdot c)\) and \((a \cdot c) + (b \cdot c) = (a + b) \cdot c\). if \(p\), \(q\), \(r\) and \(s\) were simple numbers, then making a simple pairing is easy: we can do \(e(x, y) = 2^{xy}\). then, we can see: \(e(3, 4+ 5) = 2^{3 \cdot 9} = 2^{27}\) \(e(3, 4) \cdot e(3, 5) = 2^{3 \cdot 4} \cdot 2^{3 \cdot 5} = 2^{12} \cdot 2^{15} = 2^{27}\) it's bilinear! however, such simple pairings are not suitable for cryptography because the objects that they work on are simple integers and are too easy to analyze; integers make it easy to divide, compute logarithms, and make various other computations; simple integers have no concept of a "public key" or a "one-way function". additionally, with the pairing described above you can go backwards knowing \(x\), and knowing \(e(x, y)\), you can simply compute a division and a logarithm to determine \(y\). we want mathematical objects that are as close as possible to "black boxes", where you can add, subtract, multiply and divide, but do nothing else. this is where elliptic curves and elliptic curve pairings come in. it turns out that it is possible to make a bilinear map over elliptic curve points — that is, come up with a function \(e(p, q)\) where the inputs \(p\) and \(q\) are elliptic curve points, and where the output is what's called an \((f_p)^{12}\) element (at least in the specific case we will cover here; the specifics differ depending on the details of the curve, more on this later), but the math behind doing so is quite complex. first, let's cover prime fields and extension fields. the pretty elliptic curve in the picture earlier in this post only looks that way if you assume that the curve equation is defined using regular real numbers. however, if we actually use regular real numbers in cryptography, then you can use logarithms to "go backwards", and everything breaks; additionally, the amount of space needed to actually store and represent the numbers may grow arbitrarily. hence, we instead use numbers in a prime field. a prime field consists of the set of numbers \(0, 1, 2... p-1\), where \(p\) is prime, and the various operations are defined as follows: \(a + b: (a + b)\) % \(p\) \(a \cdot b: (a \cdot b)\) % \(p\) \(a b: (a b)\) % \(p\) \(a / b: (a \cdot b^{p-2})\) % \(p\) basically, all math is done modulo \(p\) (see here for an introduction to modular math). division is a special case; normally, \(\frac{3}{2}\) is not an integer, and here we want to deal only with integers, so we instead try to find the number \(x\) such that \(x \cdot 2 = 3\), where \(\cdot\) of course refers to modular multiplication as defined above. thanks to fermat's little theorem, the exponentiation trick shown above does the job, but there is also a faster way to do it, using the extended euclidean algorithm. suppose \(p = 7\); here are a few examples: \(2 + 3 = 5\) % \(7 = 5\) \(4 + 6 = 10\) % \(7 = 3\) \(2 5 = -3\) % \(7 = 4\) \(6 \cdot 3 = 18\) % \(7 = 4\) \(3 / 2 = (3 \cdot 2^5)\) % \(7 = 5\) \(5 \cdot 2 = 10\) % \(7 = 3\) if you play around with this kind of math, you'll notice that it's perfectly consistent and satisfies all of the usual rules. the last two examples above show how \((a / b) \cdot b = a\); you can also see that \((a + b) + c = a + (b + c), (a + b) \cdot c = a \cdot c + b \cdot c\), and all the other high school algebraic identities you know and love continue to hold true as well. in elliptic curves in reality, the points and equations are usually computed in prime fields. now, let's talk about extension fields. you have probably already seen an extension field before; the most common example that you encounter in math textbooks is the field of complex numbers, where the field of real numbers is "extended" with the additional element \(\sqrt{-1} = i\). basically, extension fields work by taking an existing field, then "inventing" a new element and defining the relationship between that element and existing elements (in this case, \(i^2 + 1 = 0\)), making sure that this equation does not hold true for any number that is in the original field, and looking at the set of all linear combinations of elements of the original field and the new element that you have just created. we can do extensions of prime fields too; for example, we can extend the prime field \(\bmod 7\) that we described above with \(i\), and then we can do: \((2 + 3i) + (4 + 2i) = 6 + 5i\) \((5 + 2i) + 3 = 1 + 2i\) \((6 + 2i) \cdot 2 = 5 + 4i\) \(4i \cdot (2 + i) = 3 + i\) that last result may be a bit hard to figure out; what happened there was that we first decompose the product into \(4i \cdot 2 + 4i \cdot i\), which gives \(8i 4\), and then because we are working in \(\bmod 7\) math that becomes \(i + 3\). to divide, we do: \(a / b: (a \cdot b^{(p^2-2)})\) % \(p\) note that the exponent for fermat's little theorem is now \(p^2\) instead of \(p\), though once again if we want to be more efficient we can also instead extend the extended euclidean algorithm to do the job. note that \(x^{p^2 1} = 1\) for any \(x\) in this field, so we call \(p^2 1\) the "order of the multiplicative group in the field". with real numbers, the fundamental theorem of algebra ensures that the quadratic extension that we call the complex numbers is "complete" — you cannot extend it further, because for any mathematical relationship (at least, any mathematical relationship defined by an algebraic formula) that you can come up with between some new element \(j\) and the existing complex numbers, it's possible to come up with at least one complex number that already satisfies that relationship. with prime fields, however, we do not have this issue, and so we can go further and make cubic extensions (where the mathematical relationship between some new element \(w\) and existing field elements is a cubic equation, so \(1, w\) and \(w^2\) are all linearly independent of each other), higher-order extensions, extensions of extensions, etc. and it is these kinds of supercharged modular complex numbers that elliptic curve pairings are built on. for those interested in seeing the exact math involved in making all of these operations written out in code, prime fields and field extensions are implemented here: https://github.com/ethereum/py_pairing/blob/master/py_ecc/bn128/bn128_field_elements.py now, on to elliptic curve pairings. an elliptic curve pairing (or rather, the specific form of pairing we'll explore here; there are also other types of pairings, though their logic is fairly similar) is a map \(g_2 \times g_1 \rightarrow g_t\), where: \(\bf g_1\) is an elliptic curve, where points satisfy an equation of the form \(y^2 = x^3 + b\), and where both coordinates are elements of \(f_p\) (ie. they are simple numbers, except arithmetic is all done modulo some prime number) \(\bf g_2\) is an elliptic curve, where points satisfy the same equation as \(g_1\), except where the coordinates are elements of \((f_p)^{12}\) (ie. they are the supercharged complex numbers we talked about above; we define a new "magic number" \(w\), which is defined by a \(12\)th degree polynomial like \(w^{12} 18 \cdot w^6 + 82 = 0\)) \(\bf g_t\) is the type of object that the result of the elliptic curve goes into. in the curves that we look at, \(g_t\) is \(\bf (f_p)^{12}\) (the same supercharged complex number as used in \(g_2\)) the main property that it must satisfy is bilinearity, which in this context means that: \(e(p, q + r) = e(p, q) \cdot e(p, r)\) \(e(p + q, r) = e(p, r) \cdot e(q, r)\) there are two other important criteria: efficient computability (eg. we can make an easy pairing by simply taking the discrete logarithms of all points and multiplying them together, but this is as computationally hard as breaking elliptic curve cryptography in the first place, so it doesn't count) non-degeneracy (sure, you could just define \(e(p, q) = 1\), but that's not a particularly useful pairing) so how do we do this? the math behind why pairing functions work is quite tricky and involves quite a bit of advanced algebra going even beyond what we've seen so far, but i'll provide an outline. first of all, we need to define the concept of a divisor, basically an alternative way of representing functions on elliptic curve points. a divisor of a function basically counts the zeroes and the infinities of the function. to see what this means, let's go through a few examples. let us fix some point \(p = (p_x, p_y)\), and consider the following function: \(f(x, y) = x p_x\) the divisor is \([p] + [-p] 2 \cdot [o]\) (the square brackets are used to represent the fact that we are referring to the presence of the point \(p\) in the set of zeroes and infinities of the function, not the point p itself; \([p] + [q]\) is not the same thing as \([p + q]\)). the reasoning is as follows: the function is equal to zero at \(p\), since \(x\) is \(p_x\), so \(x p_x = 0\) the function is equal to zero at \(-p\), since \(-p\) and \(p\) share the same \(x\) coordinate the function goes to infinity as \(x\) goes to infinity, so we say the function is equal to infinity at \(o\). there's a technical reason why this infinity needs to be counted twice, so \(o\) gets added with a "multiplicity" of \(-2\) (negative because it's an infinity and not a zero, two because of this double counting). the technical reason is roughly this: because the equation of the curve is \(x^3 = y^2 + b, y\) goes to infinity "\(1.5\) times faster" than \(x\) does in order for \(y^2\) to keep up with \(x^3\); hence, if a linear function includes only \(x\) then it is represented as an infinity of multiplicity \(2\), but if it includes \(y\) then it is represented as an infinity of multiplicity \(3\). now, consider a "line function": \(ax + by + c = 0\) where \(a\), \(b\) and \(c\) are carefully chosen so that the line passes through points \(p\) and \(q\). because of how elliptic curve addition works (see the diagram at the top), this also means that it passes through \(-p-q\). and it goes up to infinity dependent on both \(x\) and \(y\), so the divisor becomes \([p]+ [q] + [-p-q] 3 \cdot [o]\). we know that every "rational function" (ie. a function defined only using a finite number of \(+, -, \cdot\) and \(/\) operations on the coordinates of the point) uniquely corresponds to some divisor, up to multiplication by a constant (ie. if two functions \(f\) and \(g\) have the same divisor, then \(f = g \cdot k\) for some constant \(k\)). for any two functions \(f\) and \(g\), the divisor of \(f \cdot g\) is equal to the divisor of \(f\) plus the divisor of \(g\) (in math textbooks, you'll see \((f \cdot g) = (f) + (g)\)), so for example if \(f(x, y) = p_x x\), then \((f^3) = 3 \cdot [p] + 3 \cdot [-p] 6 \cdot [o]\); \(p\) and \(-p\) are "triple-counted" to account for the fact that \(f^3\) approaches \(0\) at those points "three times as quickly" in a certain mathematical sense. note that there is a theorem that states that if you "remove the square brackets" from a divisor of a function, the points must add up to \(o ([p] + [q] + [-p-q] 3 \cdot [o]\) clearly fits, as \(p + q p q 3 \cdot o = o)\), and any divisor that has this property is the divisor of a function. now, we're ready to look at tate pairings. consider the following functions, defined via their divisors: \((f_p) = n \cdot [p] n \cdot [o]\), where \(n\) is the order of \(g_1\), ie. \(n \cdot p = o\) for any \(p\) \((f_q) = n \cdot [q] n \cdot [o]\) \((g) = [p + q] [p] [q] + [o]\) now, let's look at the product \(f_p \cdot f_q \cdot g^n\). the divisor is: \(n \cdot [p] n \cdot [o] + n \cdot [q] n \cdot [o] + n \cdot [p + q] n \cdot [p] n \cdot [q] + n \cdot [o]\) which simplifies neatly to: \(n \cdot [p + q] n \cdot [o]\) notice that this divisor is of exactly the same format as the divisor for \(f_p\) and \(f_q\) above. hence, \(f_p \cdot f_q \cdot g^n = f_{p + q}\). now, we introduce a procedure called the "final exponentiation" step, where we take the result of our functions above (\(f_p, f_q\), etc.) and raise it to the power \(z = (p^{12} 1) / n\), where \(p^{12} 1\) is the order of the multiplicative group in \((f_p)^{12}\) (ie. for any \(x \in (f_p)^{12}, x^{(p^{12} 1)} = 1\)). notice that if you apply this exponentiation to any result that has already been raised to the power of \(n\), you get an exponentiation to the power of \(p^{12} 1\), so the result turns into \(1\). hence, after final exponentiation, \(g^n\) cancels out and we get \(f_p^z \cdot f_q^z = (f_{p + q})^z\). there's some bilinearity for you. now, if you want to make a function that's bilinear in both arguments, you need to go into spookier math, where instead of taking \(f_p\) of a value directly, you take \(f_p\) of a divisor, and that's where the full "tate pairing" comes from. to prove some more results you have to deal with notions like "linear equivalence" and "weil reciprocity", and the rabbit hole goes on from there. you can find more reading material on all of this here and here. for an implementation of a modified version of the tate pairing, called the optimal ate paring, see here. the code implements miller's algorithm, which is needed to actually compute \(f_p\). note that the fact pairings like this are possible is somewhat of a mixed blessing: on the one hand, it means that all the protocols we can do with pairings become possible, but is also means that we have to be more careful about what elliptic curves we use. every elliptic curve has a value called an embedding degree; essentially, the smallest \(k\) such that \(p^k 1\) is a multiple of \(n\) (where \(p\) is the prime used for the field and \(n\) is the curve order). in the fields above, \(k = 12\), and in the fields used for traditional ecc (ie. where we don't care about pairings), the embedding degree is often extremely large, to the point that pairings are computationally infeasible to compute; however, if we are not careful then we can generate fields where \(k = 4\) or even \(1\). if \(k = 1\), then the "discrete logarithm" problem for elliptic curves (essentially, recovering \(p\) knowing only the point \(p = g \cdot p\), the problem that you have to solve to "crack" an elliptic curve private key) can be reduced into a similar math problem over \(f_p\), where the problem becomes much easier (this is called the mov attack); using curves with an embedding degree of \(12\) or higher ensures that this reduction is either unavailable, or that solving the discrete log problem over pairing results is at least as hard as recovering a private key from a public key "the normal way" (ie. computationally infeasible). do not worry; all standard curve parameters have been thoroughly checked for this issue. stay tuned for a mathematical explanation of how zk-snarks work, coming soon. special thanks to christian reitwiessner, ariel gabizon (from zcash) and alfred menezes for reviewing and making corrections. dark mode toggle understanding plonk 2019 sep 22 see all posts special thanks to justin drake, karl floersch, hsiao-wei wang, barry whitehat, dankrad feist, kobi gurkan and zac williamson for review very recently, ariel gabizon, zac williamson and oana ciobotaru announced a new general-purpose zero-knowledge proof scheme called plonk, standing for the unwieldy quasi-backronym "permutations over lagrange-bases for oecumenical noninteractive arguments of knowledge". while improvements to general-purpose zero-knowledge proof protocols have been coming for years, what plonk (and the earlier but more complex sonic and the more recent marlin) bring to the table is a series of enhancements that may greatly improve the usability and progress of these kinds of proofs in general. the first improvement is that while plonk still requires a trusted setup procedure similar to that needed for the snarks in zcash, it is a "universal and updateable" trusted setup. this means two things: first, instead of there being one separate trusted setup for every program you want to prove things about, there is one single trusted setup for the whole scheme after which you can use the scheme with any program (up to some maximum size chosen when making the setup). second, there is a way for multiple parties to participate in the trusted setup such that it is secure as long as any one of them is honest, and this multi-party procedure is fully sequential: first one person participates, then the second, then the third... the full set of participants does not even need to be known ahead of time; new participants could just add themselves to the end. this makes it easy for the trusted setup to have a large number of participants, making it quite safe in practice. the second improvement is that the "fancy cryptography" it relies on is one single standardized component, called a "polynomial commitment". plonk uses "kate commitments", based on a trusted setup and elliptic curve pairings, but you can instead swap it out with other schemes, such as fri (which would turn plonk into a kind of stark) or dark (based on hidden-order groups). this means the scheme is theoretically compatible with any (achievable) tradeoff between proof size and security assumptions. what this means is that use cases that require different tradeoffs between proof size and security assumptions (or developers that have different ideological positions about this question) can still share the bulk of the same tooling for "arithmetization" the process for converting a program into a set of polynomial equations that the polynomial commitments are then used to check. if this kind of scheme becomes widely adopted, we can thus expect rapid progress in improving shared arithmetization techniques. how plonk works let us start with an explanation of how plonk works, in a somewhat abstracted format that focuses on polynomial equations without immediately explaining how those equations are verified. a key ingredient in plonk, as is the case in the qaps used in snarks, is a procedure for converting a problem of the form "give me a value \(x\) such that a specific program \(p\) that i give you, when evaluated with \(x\) as an input, gives some specific result \(y\)" into the problem "give me a set of values that satisfies a set of math equations". the program \(p\) can represent many things; for example the problem could be "give me a solution to this sudoku", which you would encode by setting \(p\) to be a sudoku verifier plus some initial values encoded and setting \(y\) to \(1\) (ie. "yes, this solution is correct"), and a satisfying input \(x\) would be a valid solution to the sudoku. this is done by representing \(p\) as a circuit with logic gates for addition and multiplication, and converting it into a system of equations where the variables are the values on all the wires and there is one equation per gate (eg. \(x_6 = x_4 \cdot x_7\) for multiplication, \(x_8 = x_5 + x_9\) for addition). here is an example of the problem of finding \(x\) such that \(p(x) = x^3 + x + 5 = 35\) (hint: \(x = 3\)): we can label the gates and wires as follows: on the gates and wires, we have two types of constraints: gate constraints (equations between wires attached to the same gate, eg. \(a_1 \cdot b_1 = c_1\)) and copy constraints (claims about equality of different wires anywhere in the circuit, eg. \(a_0 = a_1 = b_1 = b_2 = a_3\) or \(c_0 = a_1\)). we will need to create a structured system of equations, which will ultimately reduce to a very small number of polynomial equations, to represent both. in plonk, the setup for these equations is as follows. each equation is of the following form (think: \(l\) = left, \(r\) = right, \(o\) = output, \(m\) = multiplication, \(c\) = constant): \[ \left(q_{l_{i}}\right) a_{i}+\left(q_{r_{i}}\right) b_{i}+\left(q_{o_{i}}\right) c_{i}+\left(q_{m_{i}}\right) a_{i} b_{i}+q_{c_{i}}=0 \] each \(q\) value is a constant; the constants in each equation (and the number of equations) will be different for each program. each small-letter value is a variable, provided by the user: \(a_i\) is the left input wire of the \(i\)'th gate, \(b_i\) is the right input wire, and \(c_i\) is the output wire of the \(i\)'th gate. for an addition gate, we set: \[ q_{l_{i}}=1, q_{r_{i}}=1, q_{m_{i}}=0, q_{o_{i}}=-1, q_{c_{i}}=0 \] plugging these constants into the equation and simplifying gives us \(a_i + b_i c_i = 0\), which is exactly the constraint that we want. for a multiplication gate, we set: \[ q_{l_{i}}=0, q_{r_{i}}=0, q_{m_{i}}=1, q_{o_{i}}=-1, q_{c_{i}}=0 \] for a constant gate setting \(a_i\) to some constant \(x\), we set: \[ q_{l}=1, q_{r}=0, q_{m}=0, q_{o}=0, q_{c}=-x \] you may have noticed that each end of a wire, as well as each wire in a set of wires that clearly must have the same value (eg. \(x\)), corresponds to a distinct variable; there's nothing so far forcing the output of one gate to be the same as the input of another gate (what we call "copy constraints"). plonk does of course have a way of enforcing copy constraints, but we'll get to this later. so now we have a problem where a prover wants to prove that they have a bunch of \(x_{a_i}, x_{b_i}\) and \(x_{c_i}\) values that satisfy a bunch of equations that are of the same form. this is still a big problem, but unlike "find a satisfying input to this computer program" it's a very structured big problem, and we have mathematical tools to "compress" it. from linear systems to polynomials if you have read about starks or qaps, the mechanism described in this next section will hopefully feel somewhat familiar, but if you have not that's okay too. the main ingredient here is to understand a polynomial as a mathematical tool for encapsulating a whole lot of values into a single object. typically, we think of polynomials in "coefficient form", that is an expression like: \[ y=x^{3}-5 x^{2}+7 x-2 \] but we can also view polynomials in "evaluation form". for example, we can think of the above as being "the" degree \(< 4\) polynomial with evaluations \((-2, 1, 0, 1)\) at the coordinates \((0, 1, 2, 3)\) respectively. now here's the next step. systems of many equations of the same form can be re-interpreted as a single equation over polynomials. for example, suppose that we have the system: \[ \begin{array}{l}{2 x_{1}-x_{2}+3 x_{3}=8} \\ {x_{1}+4 x_{2}-5 x_{3}=5} \\ {8 x_{1}-x_{2}-x_{3}=-2}\end{array} \] let us define four polynomials in evaluation form: \(l(x)\) is the degree \(< 3\) polynomial that evaluates to \((2, 1, 8)\) at the coordinates \((0, 1, 2)\), and at those same coordinates \(m(x)\) evaluates to \((-1, 4, -1)\), \(r(x)\) to \((3, -5, -1)\) and \(o(x)\) to \((8, 5, -2)\) (it is okay to directly define polynomials in this way; you can use lagrange interpolation to convert to coefficient form). now, consider the equation: \[ l(x) \cdot x_{1}+m(x) \cdot x_{2}+r(x) \cdot x_{3}-o(x)=z(x) h(x) \] here, \(z(x)\) is shorthand for \((x-0) \cdot (x-1) \cdot (x-2)\) the minimal (nontrivial) polynomial that returns zero over the evaluation domain \((0, 1, 2)\). a solution to this equation (\(x_1 = 1, x_2 = 6, x_3 = 4, h(x) = 0\)) is also a solution to the original system of equations, except the original system does not need \(h(x)\). notice also that in this case, \(h(x)\) is conveniently zero, but in more complex cases \(h\) may need to be nonzero. so now we know that we can represent a large set of constraints within a small number of mathematical objects (the polynomials). but in the equations that we made above to represent the gate wire constraints, the \(x_1, x_2, x_3\) variables are different per equation. we can handle this by making the variables themselves polynomials rather than constants in the same way. and so we get: \[ q_{l}(x) a(x)+q_{r}(x) b(x)+q_{o}(x) c(x)+q_{m}(x) a(x) b(x)+q_{c}(x)=0 \] as before, each \(q\) polynomial is a parameter that can be generated from the program that is being verified, and the \(a\), \(b\), \(c\) polynomials are the user-provided inputs. copy constraints now, let us get back to "connecting" the wires. so far, all we have is a bunch of disjoint equations about disjoint values that are independently easy to satisfy: constant gates can be satisfied by setting the value to the constant and addition and multiplication gates can simply be satisfied by setting all wires to zero! to make the problem actually challenging (and actually represent the problem encoded in the original circuit), we need to add an equation that verifies "copy constraints": constraints such as \(a(5) = c(7)\), \(c(10) = c(12)\), etc. this requires some clever trickery. our strategy will be to design a "coordinate pair accumulator", a polynomial \(p(x)\) which works as follows. first, let \(x(x)\) and \(y(x)\) be two polynomials representing the \(x\) and \(y\) coordinates of a set of points (eg. to represent the set \(((0, -2), (1, 1), (2, 0), (3, 1))\) you might set \(x(x) = x\) and \(y(x) = x^3 5x^2 + 7x 2)\). our goal will be to let \(p(x)\) represent all the points up to (but not including) the given position, so \(p(0)\) starts at \(1\), \(p(1)\) represents just the first point, \(p(2)\) the first and the second, etc. we will do this by "randomly" selecting two constants, \(v_1\) and \(v_2\), and constructing \(p(x)\) using the constraints \(p(0) = 1\) and \(p(x+1) = p(x) \cdot (v_1 + x(x) + v_2 \cdot y(x))\) at least within the domain \((0, 1, 2, 3)\). for example, letting \(v_1 = 3\) and \(v_2 = 2\), we get: x(x) 0 1 2 3 4 \(y(x)\) -2 1 0 1 \(v_1 + x(x) + v_2 \cdot y(x)\) -1 6 5 8 \(p(x)\) 1 -1 -6 -30 -240 notice that (aside from the first column) every \(p(x)\) value equals the value to the left of it multiplied by the value to the left and above it. the result we care about is \(p(4) = -240\). now, consider the case where instead of \(x(x) = x\), we set \(x(x) = \frac{2}{3} x^3 4x^2 + \frac{19}{3}x\) (that is, the polynomial that evaluates to \((0, 3, 2, 1)\) at the coordinates \((0, 1, 2, 3)\)). if you run the same procedure, you'll find that you also get \(p(4) = -240\). this is not a coincidence (in fact, if you randomly pick \(v_1\) and \(v_2\) from a sufficiently large field, it will almost never happen coincidentally). rather, this happens because \(y(1) = y(3)\), so if you "swap the \(x\) coordinates" of the points \((1, 1)\) and \((3, 1)\) you're not changing the set of points, and because the accumulator encodes a set (as multiplication does not care about order) the value at the end will be the same. now we can start to see the basic technique that we will use to prove copy constraints. first, consider the simple case where we only want to prove copy constraints within one set of wires (eg. we want to prove \(a(1) = a(3)\)). we'll make two coordinate accumulators: one where \(x(x) = x\) and \(y(x) = a(x)\), and the other where \(y(x) = a(x)\) but \(x'(x)\) is the polynomial that evaluates to the permutation that flips (or otherwise rearranges) the values in each copy constraint; in the \(a(1) = a(3)\) case this would mean the permutation would start \(0 3 2 1 4...\). the first accumulator would be compressing \(((0, a(0)), (1, a(1)), (2, a(2)), (3, a(3)), (4, a(4))...\), the second \(((0, a(0)), (3, a(1)), (2, a(2)), (1, a(3)), (4, a(4))...\). the only way the two can give the same result is if \(a(1) = a(3)\). to prove constraints between \(a\), \(b\) and \(c\), we use the same procedure, but instead "accumulate" together points from all three polynomials. we assign each of \(a\), \(b\), \(c\) a range of \(x\) coordinates (eg. \(a\) gets \(x_a(x) = x\) ie. \(0...n-1\), \(b\) gets \(x_b(x) = n+x\), ie. \(n...2n-1\), \(c\) gets \(x_c(x) = 2n+x\), ie. \(2n...3n-1\). to prove copy constraints that hop between different sets of wires, the "alternate" \(x\) coordinates would be slices of a permutation across all three sets. for example, if we want to prove \(a(2) = b(4)\) with \(n = 5\), then \(x'_a(x)\) would have evaluations \(\{0, 1, 9, 3, 4\}\) and \(x'_b(x)\) would have evaluations \(\{5, 6, 7, 8, 2\}\) (notice the \(2\) and \(9\) flipped, where \(9\) corresponds to the \(b_4\) wire). often, \(x'_a(x)\), \(x'_b(x)\) and \(x'_c(x)\) are also called \(\sigma_a(x)\), \(\sigma_b(x)\) and \(\sigma_c(x)\). we would then instead of checking equality within one run of the procedure (ie. checking \(p(4) = p'(4)\) as before), we would check the product of the three different runs on each side: \[ p_{a}(n) \cdot p_{b}(n) \cdot p_{c}(n)=p_{a}^{\prime}(n) \cdot p_{b}^{\prime}(n) \cdot p_{c}^{\prime}(n) \] the product of the three \(p(n)\) evaluations on each side accumulates all coordinate pairs in the \(a\), \(b\) and \(c\) runs on each side together, so this allows us to do the same check as before, except that we can now check copy constraints not just between positions within one of the three sets of wires \(a\), \(b\) or \(c\), but also between one set of wires and another (eg. as in \(a(2) = b(4)\)). and that's all there is to it! putting it all together in reality, all of this math is done not over integers, but over a prime field; check the section "a modular math interlude" here for a description of what prime fields are. also, for mathematical reasons perhaps best appreciated by reading and understanding this article on fft implementation, instead of representing wire indices with \(x=0....n-1\), we'll use powers of \(\omega: 1, \omega, \omega ^2....\omega ^{n-1}\) where \(\omega\) is a high-order root-of-unity in the field. this changes nothing about the math, except that the coordinate pair accumulator constraint checking equation changes from \(p(x + 1) = p(x) \cdot (v_1 + x(x) + v_2 \cdot y(x))\) to \(p(\omega \cdot x) = p(x) \cdot (v_1 + x(x) + v_2 \cdot y(x))\), and instead of using \(0..n-1\), \(n..2n-1\), \(2n..3n-1\) as coordinates we use \(\omega^i, g \cdot \omega^i\) and \(g^2 \cdot \omega^i\) where \(g\) can be some random high-order element in the field. now let's write out all the equations we need to check. first, the main gate-constraint satisfaction check: \[ q_{l}(x) a(x)+q_{r}(x) b(x)+q_{o}(x) c(x)+q_{m}(x) a(x) b(x)+q_{c}(x)=0 \] then the polynomial accumulator transition constraint (note: think of "\(= z(x) \cdot h(x)\)" as meaning "equals zero for all coordinates within some particular domain that we care about, but not necessarily outside of it"): \[ \begin{array}{l}{p_{a}(\omega x)-p_{a}(x)\left(v_{1}+x+v_{2} a(x)\right) =z(x) h_{1}(x)} \\ {p_{a^{\prime}}(\omega x)-p_{a^{\prime}}(x)\left(v_{1}+\sigma_{a}(x)+v_{2} a(x)\right)=z(x) h_{2}(x)} \\ {p_{b}(\omega x)-p_{b}(x)\left(v_{1}+g x+v_{2} b(x)\right)=z(x) h_{3}(x)} \\ {p_{b^{\prime}}(\omega x)-p_{b^{\prime}}(x)\left(v_{1}+\sigma_{b}(x)+v_{2} b(x)\right)=z(x) h_{4}(x)} \\ {p_{c}(\omega x)-p_{c}(x)\left(v_{1}+g^{2} x+v_{2} c(x)\right)=z(x) h_{5}(x)} \\ {p_{c^{\prime}}(\omega x)-p_{c^{\prime}}(x)\left(v_{1}+\sigma_{c}(x)+v_{2} c(x)\right)=z(x) h_{6}(x)}\end{array} \] then the polynomial accumulator starting and ending constraints: \[ \begin{array}{l}{p_{a}(1)=p_{b}(1)=p_{c}(1)=p_{a^{\prime}}(1)=p_{b^{\prime}}(1)=p_{c^{\prime}}(1)=1} \\ {p_{a}\left(\omega^{n}\right) p_{b}\left(\omega^{n}\right) p_{c}\left(\omega^{n}\right)=p_{a^{\prime}}\left(\omega^{n}\right) p_{b^{\prime}}\left(\omega^{n}\right) p_{c^{\prime}}\left(\omega^{n}\right)}\end{array} \] the user-provided polynomials are: the wire assignments \(a(x), b(x), c(x)\) the coordinate accumulators \(p_a(x), p_b(x), p_c(x), p_{a'}(x), p_{b'}(x), p_{c'}(x)\) the quotients \(h(x)\) and \(h_1(x)...h_6(x)\) the program-specific polynomials that the prover and verifier need to compute ahead of time are: \(q_l(x), q_r(x), q_o(x), q_m(x), q_c(x)\), which together represent the gates in the circuit (note that \(q_c(x)\) encodes public inputs, so it may need to be computed or modified at runtime) the "permutation polynomials" \(\sigma_a(x), \sigma_b(x)\) and \(\sigma_c(x)\), which encode the copy constraints between the \(a\), \(b\) and \(c\) wires note that the verifier need only store commitments to these polynomials. the only remaining polynomial in the above equations is \(z(x) = (x 1) \cdot (x \omega) \cdot ... \cdot (x \omega ^{n-1})\) which is designed to evaluate to zero at all those points. fortunately, \(\omega\) can be chosen to make this polynomial very easy to evaluate: the usual technique is to choose \(\omega\) to satisfy \(\omega ^n = 1\), in which case \(z(x) = x^n 1\). there is one nuance here: the constraint between \(p_a(\omega^{i+1})\) and \(p_a(\omega^i)\) can't be true across the entire circle of powers of \(\omega\); it's almost always false at \(\omega^{n-1}\) as the next coordinate is \(\omega^n = 1\) which brings us back to the start of the "accumulator"; to fix this, we can modify the constraint to say "either the constraint is true or \(x = \omega^{n-1}\)", which one can do by multiplying \(x \omega^{n-1}\) into the constraint so it equals zero at that point. the only constraint on \(v_1\) and \(v_2\) is that the user must not be able to choose \(a(x), b(x)\) or \(c(x)\) after \(v_1\) and \(v_2\) become known, so we can satisfy this by computing \(v_1\) and \(v_2\) from hashes of commitments to \(a(x), b(x)\) and \(c(x)\). so now we've turned the program satisfaction problem into a simple problem of satisfying a few equations with polynomials, and there are some optimizations in plonk that allow us to remove many of the polynomials in the above equations that i will not go into to preserve simplicity. but the polynomials themselves, both the program-specific parameters and the user inputs, are big. so the next question is, how do we get around this so we can make the proof short? polynomial commitments a polynomial commitment is a short object that "represents" a polynomial, and allows you to verify evaluations of that polynomial, without needing to actually contain all of the data in the polynomial. that is, if someone gives you a commitment \(c\) representing \(p(x)\), they can give you a proof that can convince you, for some specific \(z\), what the value of \(p(z)\) is. there is a further mathematical result that says that, over a sufficiently big field, if certain kinds of equations (chosen before \(z\) is known) about polynomials evaluated at a random \(z\) are true, those same equations are true about the whole polynomial as well. for example, if \(p(z) \cdot q(z) + r(z) = s(z) + 5\), then we know that it's overwhelmingly likely that \(p(x) \cdot q(x) + r(x) = s(x) + 5\) in general. using such polynomial commitments, we could very easily check all of the above polynomial equations above make the commitments, use them as input to generate \(z\), prove what the evaluations are of each polynomial at \(z\), and then run the equations with these evaluations instead of the original polynomials. but how do these commitments work? there are two parts: the commitment to the polynomial \(p(x) \rightarrow c\), and the opening to a value \(p(z)\) at some \(z\). to make a commitment, there are many techniques; one example is fri, and another is kate commitments which i will describe below. to prove an opening, it turns out that there is a simple generic "subtract-and-divide" trick: to prove that \(p(z) = a\), you prove that \[ \frac{p(x)-a}{x-z} \] is also a polynomial (using another polynomial commitment). this works because if the quotient is a polynomial (ie. it is not fractional), then \(x z\) is a factor of \(p(x) a\), so \((p(x) a)(z) = 0\), so \(p(z) = a\). try it with some polynomial, eg. \(p(x) = x^3 + 2 \cdot x^2 + 5\) with \((z = 6, a = 293)\), yourself; and try \((z = 6, a = 292)\) and see how it fails (if you're lazy, see wolframalpha here vs here). note also a generic optimization: to prove many openings of many polynomials at the same time, after committing to the outputs do the subtract-and-divide trick on a random linear combination of the polynomials and the outputs. so how do the commitments themselves work? kate commitments are, fortunately, much simpler than fri. a trusted-setup procedure generates a set of elliptic curve points \(g, g \cdot s, g \cdot s^2\) .... \(g \cdot s^n\), as well as \(g_2 \cdot s\), where \(g\) and \(g_2\) are the generators of two elliptic curve groups and \(s\) is a secret that is forgotten once the procedure is finished (note that there is a multi-party version of this setup, which is secure as long as at least one of the participants forgets their share of the secret). these points are published and considered to be "the proving key" of the scheme; anyone who needs to make a polynomial commitment will need to use these points. a commitment to a degree-d polynomial is made by multiplying each of the first d+1 points in the proving key by the corresponding coefficient in the polynomial, and adding the results together. notice that this provides an "evaluation" of that polynomial at \(s\), without knowing \(s\). for example, \(x^3 + 2x^2+5\) would be represented by \((g \cdot s^3) + 2 \cdot (g \cdot s^2) + 5 \cdot g\). we can use the notation \([p]\) to refer to \(p\) encoded in this way (ie. \(g \cdot p(s)\)). when doing the subtract-and-divide trick, you can prove that the two polynomials actually satisfy the relation by using elliptic curve pairings: check that \(e([p] g \cdot a, g_2) = e([q], [x] g_2 \cdot z)\) as a proxy for checking that \(p(x) a = q(x) \cdot (x z)\). but there are more recently other types of polynomial commitments coming out too. a new scheme called dark ("diophantine arguments of knowledge") uses "hidden order groups" such as class groups to implement another kind of polynomial commitment. hidden order groups are unique because they allow you to compress arbitrarily large numbers into group elements, even numbers much larger than the size of the group element, in a way that can't be "spoofed"; constructions from vdfs to accumulators to range proofs to polynomial commitments can be built on top of this. another option is to use bulletproofs, using regular elliptic curve groups at the cost of the proof taking much longer to verify. because polynomial commitments are much simpler than full-on zero knowledge proof schemes, we can expect more such schemes to get created in the future. recap to finish off, let's go over the scheme again. given a program \(p\), you convert it into a circuit, and generate a set of equations that look like this: \[ \left(q_{l_{i}}\right) a_{i}+\left(q_{r_{i}}\right) b_{i}+\left(q_{o_{i}}\right) c_{i}+\left(q_{m_{i}}\right) a_{i} b_{i}+q_{c_{i}}=0 \] you then convert this set of equations into a single polynomial equation: \[ q_{l}(x) a(x)+q_{r}(x) b(x)+q_{o}(x) c(x)+q_{m}(x) a(x) b(x)+q_{c}(x)=0 \] you also generate from the circuit a list of copy constraints. from these copy constraints you generate the three polynomials representing the permuted wire indices: \(\sigma_a(x), \sigma_b(x), \sigma_c(x)\). to generate a proof, you compute the values of all the wires and convert them into three polynomials: \(a(x), b(x), c(x)\). you also compute six "coordinate pair accumulator" polynomials as part of the permutation-check argument. finally you compute the cofactors \(h_i(x)\). there is a set of equations between the polynomials that need to be checked; you can do this by making commitments to the polynomials, opening them at some random \(z\) (along with proofs that the openings are correct), and running the equations on these evaluations instead of the original polynomials. the proof itself is just a few commitments and openings and can be checked with a few equations. and that's all there is to it! scalability, part 2: hypercubes | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search scalability, part 2: hypercubes posted by vitalik buterin on october 21, 2014 research & development special thanks to vlad zamfir, chris barnett and dominic williams for ideas and inspiration in a recent blog post i outlined some partial solutions to scalability, all of which fit into the umbrella of ethereum 1.0 as it stands. specialized micropayment protocols such as channels and probabilistic payment systems could be used to make small payments, using the blockchain either only for eventual settlement, or only probabilistically. for some computation-heavy applications, computation can be done by one party by default, but in a way that can be "pulled down" to be audited by the entire chain if someone suspects malfeasance. however, these approaches are all necessarily application-specific, and far from ideal. in this post, i describe a more comprehensive approach, which, while coming at the cost of some "fragility" concerns, does provide a solution which is much closer to being universal. understanding the objective first of all, before we get into the details, we need to get a much deeper understanding of what we actually want. what do we mean by scalability, particularly in an ethereum context? in the context of a bitcoin-like currency, the answer is relatively simple; we want to be able to: process tens of thousands of transactions per second provide a transaction fee of less than $0.001 do it all while maintaining security against at least 25% attacks and without highly centralized full nodes the first goal alone is easy; we just remove the block size limit and let the blockchain naturally grow until it becomes that large, and the economy takes care of itself to force smaller full nodes to continue to drop out until the only three full nodes left are run by ghash.io, coinbase and circle. at that point, some balance will emerge between fees and size, as excessize size leads to more centralization which leads to more fees due to monopoly pricing. in order to achieve the second, we can simply have many altcoins. to achieve all three combined, however, we need to break through a fundamental barrier posed by bitcoin and all other existing cryptocurrencies, and create a system that works without the existence of any "full nodes" that need to process every transaction. in an ethereum context, the definition of scalability gets a little more complicated. ethereum is, fundamentally, a platform for "dapps", and within that mandate there are two kinds of scalability that are relevant: allow lots and lots of people to build dapps, and keep the transaction fees low allow each individual dapp to be scalable according to a definition similar to that for bitcoin the first is inherently easier than the second. the only property that the "build lots and lots of alt-etherea" approach does not have is that each individual alt-ethereum has relatively weak security; at a size of 1000 alt-etherea, each one would be vulnerable to a 0.1% attack from the point of view of the whole system (that 0.1% is for externally-sourced attacks; internally-sourced attacks, the equivalent of ghash.io and discus fish colluding, would take only 0.05%). if we can find some way for all alt-etherea to share consensus strength, eg. some version of merged mining that makes each chain receive the strength of the entire pack without requiring the existence of miners that know about all chains simultaneously, then we would be done. the second is more problematic, because it leads to the same fragility property that arises from scaling bitcoin the currency: if every node sees only a small part of the state, and arbitrary amounts of btc can legitimately appear in any part of the state originating from any part of the state (such fungibility is part of the definition of a currency), then one can intuitively see how forgery attacks might spread through the blockchain undetected until it is too late to revert everything without substantial system-wide disruption via a global revert. reinventing the wheel we'll start off by describing a relatively simple model that does provide both kinds of scalability, but provides the second only in a very weak and costly way; essentially, we have just enough intra-dapp scalability to ensure asset fungibility, but not much more. the model works as follows: suppose that the global ethereum state (ie. all accounts, contracts and balances) is split up into n parts ("substates") (think 10 <= n <= 200). anyone can set up an account on any substate, and one can send a transaction to any substate by adding a substate number flag to it, but ordinary transactions can only send a message to an account in the same substate as the sender. however, to ensure security and cross-transmissibility, we add some more features. first, there is also a special "hub substate", which contains only a list of messages, of the form [dest_substate, address, value, data]. second, there is an opcode cross_send, which takes those four parameters as arguments, and sends such a one-way message enroute to the destination substate. miners mine blocks on some substate s[j], and each block on s[j] is simultaneously a block in the hub chain. each block on s[j] has as dependencies the previous block on s[j] and the previous block on the hub chain. for example, with n = 2, the chain would look something like this: the block-level state transition function, if mining on substate s[j], does three things: processes state transitions inside of s[j] if any of those state transitions creates a cross_send, adds that message to the hub chain if any messages are on the hub chain with dest_substate = j, removes the messages from the hub chain, sends the messages to their destination addresses on s[j], and processes all resulting state transitions from a scalability perspective, this gives us a substantial improvement. all miners only need to be aware of two out of the total n + 1 substates: their own substate, and the hub substate. dapps that are small and self-contained will exist on one substate, and dapps that want to exist across multiple substates will need to send messages through the hub. for example a cross-substate currency dapp would maintain a contract on all substates, and each contract would have an api that allows a user to destroy currency units inside of one substate in exchange for the contract sending a message that would lead to the user being credited the same amount on another substate. messages going through the hub do need to be seen by every node, so these will be expensive; however, in the case of ether or sub-currencies we only need the transfer mechanism to be used occasionally for settlement, doing off-chain inter-substate exchange for most transfers. attacks, challenges and responses now, let us take this simple scheme and analyze its security properties (for illustrative purposes, we'll use n = 100). first of all, the scheme is secure against double-spend attacks up to 50% of the total hashpower; the reason is that every sub-chain is essentially merge-mined with every other sub-chain, with each block reinforcing the security of all sub-chains simultaneously. however, there are more dangerous classes of attacks as well. suppose that a hostile attacker with 4% hashpower jumps onto one of the substates, thereby now comprising 80% of the mining power on it. now, that attacker mines blocks that are invalid for example, the attacker includes a state transition that creates messages sending 1000000 eth to every other substate out of nowhere. other miners on the same substate will recognize the hostile miner's blocks as invalid, but this is irrelevant; they are only a very small part of the total network, and only 20% of that substate. the miners on other substates don't know that the attacker's blocks are invalid, because they have no knowledge of the state of the "captured substate", so at first glance it seems as though they might blindly accept them. fortunately, here the solution here is more complex, but still well within the reach of what we currently know works: as soon as one of the few legitimate miners on the captured substate processes the invalid block, they will see that it's invalid, and therefore that it's invalid in some particular place. from there, they will be able to create a light-client merkle tree proof showing that that particular part of the state transition was invalid. to explain how this works in some detail, a light client proof consists of three things: the intermediate state root that the state transition started from the intermediate state root that the state transition ended at the subset of patricia tree nodes that are accessed or modified in the process of executing the state transition the first two "intermediate state roots" are the roots of the ethereum patricia state tree before and after executing the transaction; the ethereum protocol requires both of these to be in every block. the patricia state tree nodes provided are needed in order to the verifier to follow along the computation themselves, and see that the same result is arrived at the end. for example, if a transaction ends up modifying the state of three accounts, the set of tree nodes that will need to be provided might look something like this: technically, the proof should include the set of patricia tree nodes that are needed to access the intermediate state roots and the transaction as well, but that's a relatively minor detail. altogether, one can think of the proof as consisting of the minimal amount of information from the blockchain needed to process that particular transaction, plus some extra nodes to prove that those bits of the blockchain are actually in the current state. once the whistleblower creates this proof, they will then be broadcasted to the network, and all other miners will see the proof and discard the defective block. the hardest class of attack of all, however, is what is called a "data unavailability attack". here, imagine that the miner sends out only the block header to the network, as well as the list of messages to add to the hub, but does not provide any of the transactions, intermediate state roots or anything else. now, we have a problem. theoretically, it is entirely possible that the block is completely legitimate; the block could have been properly constructed by gathering some transactions from a few millionaires who happened to be really generous. in reality, of course, this is not the case, and the block is a fraud, but the fact that the data is not available at all makes it impossible to construct an affirmative proof of the fraud. the 20% honest miners on the captured substate may yell and squeal, but they have no proof at all, and any protocol that did heed their words would necessarily fall to a 0.2% denial-of-service attack where the miner captures 20% of a substate and pretends that the other 80% of miners on that substate are conspiring against him. to solve this problem, we need something called a challenge-response protocol. essentially, the mechanism works as follows: honest miners on the captured substate see the header-only block. an honest miner sends out a "challenge" in the form of an index (ie. a number). if the producer of the block can submit a "response" to the challenge, consisting of a light-client proof that the transaction execution at the given index was executed legitimately (or a proof that the given index is greater than the number of transactions in the block), then the challenge is deemed answered. if a challenge goes unanswered for a few seconds, miners on other substates consider the block suspicious and refuse to mine on it (the game-theoretic justification for why is the same as always: because they suspect that others will use the same strategy, and there is no point mining on a substate that will soon be orphaned) note that the mechanism requires a few added complexities on order to work. if a block is published alongside all of its transactions except for a few, then the challenge-response protocol could quickly go through them all and discard the block. however, if a block was published truly headers-only, then if the block contained hundreds of transactions, hundreds of challenges would be required. one heuristic approach to solving the problem is that miners receiving a block should privately pick some random nonces, send out a few challenges for those nonces to some known miners on the potentially captured substate, and if responses to all challenges do not come back immediately treat the block as suspect. note that the miner does not broadcast the challenge publicly that would give an opportunity for an attacker to quickly fill in the missing data. the second problem is that the protocol is vulnerable to a denial-of-service attack consisting of attackers publishing very very many challenges to legitimate blocks. to solve this, making a challenge should have some cost however, if this cost is too high then the act of making a challenge will require a very high "altruism delta", perhaps so high that an attack will eventually come and no one will challenge it. although some may be inclined to solve this with a market-based approach that places responsibility for making the challenge on whatever parties end up robbed by the invalid state transition, it is worth noting that it's possible to come up with a state transition that generates new funds out of nowhere, stealing from everyone very slightly via inflation, and also compensates wealthy coin holders, creating a theft where there is no concentrated incentive to challenge it. for a currency, one "easy solution" is capping the value of a transaction, making the entire problem have only very limited consequence. for a turing-complete protocol the solution is more complex; the best approaches likely involve both making challenges expensive and adding a mining reward to them. there will be a specialized group of "challenge miners", and the theory is that they will be indifferent as to which challenges to make, so even the tiniest altruism delta, enforced by software defaults, will drive them to make correct challenges. one may even try to measure how long challenges take to get responded, and more highly reward the ones that take longer. the twelve-dimensional hypercube note: this is not the same as the erasure-coding borg cube. for more info on that, see here: https://blog.ethereum.org/2014/08/16/secret-sharing-erasure-coding-guide-aspiring-dropbox-decentralizer/ we can see two flaws in the above scheme. first, the justification that the challenge-response protocol will work is rather iffy at best, and has poor degenerate-case behavior: a substate takeover attack combined with a denial of service attack preventing challenges could potentially force an invalid block into a chain, requiring an eventual day-long revert of the entire chain when (if?) the smoke clears. there is also a fragility component here: an invalid block in any substate will invalidate all subsequent blocks in all substates. second, cross-substate messages must still be seen by all nodes. we start off by solving the second problem, then proceed to show a possible defense to make the first problem slightly less bad, and then finally get around to solving it completely, and at the same time getting rid of proof of work. the second flaw, the expensiveness of cross-substate messages, we solve by converting the blockchain model from this: to this: except the cube should have twelve dimensions instead of three. now, the protocol looks as follows: there exist 2n substates, each of which is identified by a binary string of length n (eg. 0010111111101). we define the hamming distance h(s1, s2) as the number of digits that are different between the ids of substates s1 and s2 (eg. hd(00110, 00111) = 1, hd(00110, 10010) = 2, etc). the state of each substate stores the ordinary state tree as before, but also an outbox. there exists an opcode, cross_send, which takes 4 arguments [dest_substate, to_address, value, data], and registers a message with those arguments in the outbox of s_from where s_from is the substate from which the opcode was called all miners must "mine an edge"; that is, valid blocks are blocks which modify two adjacent substates s_a and s_b, and can include transactions for either substate. the block-level state transition function is as follows: process all transactions in order, applying the state transitions to s_a or s_b as needed. process all messages in the outboxes of s_a and s_b in order. if the message is in the outbox of s_a and has final destination s_b, process the state transitions, and likewise for messages from s_b to s_a. otherwise, if a message is in s_a and hd(s_b, msg.dest) < hd(s_a, msg.dest), move the message from the outbox of s_a to the outbox of s_b, and likewise vice versa. there exists a header chain keeping track of all headers, allowing all of these blocks to be merge-mined, and keeping one centralized location where the roots of each state are stored. essentially, instead of travelling through the hub, messages make their way around the substates along edges, and the constantly reducing hamming distance ensures that each message always eventually gets to its destination. the key design decision here is the arrangement of all substates into a hypercube. why was the cube chosen? the best way to think of the cube is as a compromise between two extreme options: on the one hand the circle, and on the other hand the simplex (basically, 2n-dimensional version of a tetrahedron). in a circle, a message would need to travel on average a quarter of the way across the circle before it gets to its destination, meaning that we make no efficiency gains over the plain old hub-and-spoke model. in a simplex, every pair of substates has an edge, so a cross-substate message would get across as soon as a block between those two substates is produced. however, with miners picking random edges it would take a long time for a block on the right edge to appear, and more importantly users watching a particular substate would need to be at least light clients on every other substate in order to validate blocks that are relevant to them. the hypercube is a perfect balance each substate has a logarithmically growing number of neighbors, the length of the longest path grows logarithmically, and block time of any particular edge grows logarithmically. note that this algorithm has essentially the same flaws as the hub-and-spoke approach namely, that it has bad degenerate-case behavior and the economics of challenge-response protocols are very unclear. to add stability, one approach is to modify the header chain somewhat. right now, the header chain is very strict in its validity requirements if any block anywhere down the header chain turns out to be invalid, all blocks in all substates on top of that are invalid and must be redone. to mitigate this, we can require the header chain to simply keep track of headers, so it can contain both invalid headers and even multiple forks of the same substate chain. to add a merge-mining protocol, we implement exponential subjective scoring but using the header chain as an absolute common timekeeper. we use a low base (eg. 0.75 instead of 0.99) and have a maximum penalty factor of 1 / 2n to remove the benefit from forking the header chain; for those not well versed in the mechanics of ess, this basically means "allow the header chain to contain all headers, but use the ordering of the header chain to penalize blocks that come later without making this penalty too strict". then, we add a delay on cross-substate messages, so a message in an outbox only becomes "eligible" if the originating block is at least a few dozen blocks deep. proof of stake now, let us work on porting the protocol to nearly-pure proof of stake. we'll ignore nothing-at-stake issues for now; slasher-like protocols plus exponential subjective scoring can solve those concerns, and we will discuss adding them in later. initially, our objective is to show how to make the hypercube work without mining, and at the same time partially solve the fragility problem. we will start off with a proof of activity implementation for multichain. the protocol works as follows: there exist 2n substates indentified by binary string, as before, as well as a header chain (which also keeps track of the latest state root of each substate). anyone can mine an edge, as before, but with a lower difficulty. however, when a block is mined, it must be published alongside the complete set of merkle tree proofs so that a node with no prior information can fully validate all state transitions in the block. there exists a bonding protocol where an address can specify itself as a potential signer by submitting a bond of size b (richer addresses will need to create multiple sub-accounts). potential signers are stored in a specialized contract c[s] on each substate s. based on the block hash, a random 200 substates s[i] are chosen, and a search index 0 <= ind[i] < 2^160 is chosen for each substate. define signer[i] as the owner of the first address in c[s[i]] after index ind[i]. for the block to be valid, it must be signed by at least 133 of the set signer[0] ... signer[199]. to actually check the validity of a block, the consensus group members would do two things. first, they would check that the initial state roots provided in the block match the corresponding state roots in the header chain. second, they would process the transactions, and make sure that the final state roots match the final state roots provided in the header chain and that all trie nodes needed to calculate the update are available somewhere in the network. if both checks pass, they sign the block, and if the block is signed by sufficiently many consensus group members it gets added to the header chain, and the state roots for the two affected blocks in the header chain are updated. and that's all there is to it. the key property here is that every block has a randomly chosen consensus group, and that group is chosen from the global state of all account holders. hence, unless an attacker has at least 33% of the stake in the entire system, it will be virtually impossible (specifically, 2-70 probability, which with 230 proof of work falls well into the realm of cryptographic impossiblity) for the attacker to get a block signed. and without 33% of the stake, an attacker will not be able to prevent legitimate miners from creating blocks and getting them signed. this approach has the benefit that it has nice degenerate-case behavior; if a denial-of-service attack happens, then chances are that almost no blocks will be produced, or at least blocks will be produced very slowly, but no damage will be done. now, the challenge is, how do we further reduce proof of work dependence, and add in blockmaker and slasher-based protocols? a simple approach is to have a separate blockmaker protocol for every edge, just as in the single-chain approach. to incentivize blockmakers to act honestly and not double-sign, slasher can also be used here: if a signer signs a block that ends up not being in the main chain, they get punished. schelling point effects ensure that everyone has the incentive to follow the protocol, as they guess that everyone else will (with the additional minor pseudo-incentive of software defaults to make the equilibrium stronger). a full evm these protocols allow us to send one-way messages from one substate to another. however, one way messages are limited in functionality (or rather, they have as much functionality as we want them to have because everything is turing-complete, but they are not always the nicest to work with). what if we can make the hypercube simulate a full cross-substate evm, so you can even call functions that are on other substates? as it turns out, you can. the key is to add to messages a data structure called a continuation. for example, suppose that we are in the middle of a computation where a contract calls a contract which creates a contract, and we are currently executing the code that is creating the inner contract. thus, the place we are in the computation looks something like this: now, what is the current "state" of this computation? that is, what is the set of all the data that we need to be able to pause the computation, and then using the data resume it later on? in a single instance of the evm, that's just the program counter (ie. where we are in the code), the memory and the stack. in a situation with contracts calling each other, we need that data for the entire "computational tree", including where we are in the current scope, the parent scope, the parent of that, and so forth back to the original transaction: this is called a "continuation". to resume an execution from this continuation, we simply resume each computation and run it to completion in reverse order (ie. finish the innermost first, then put its output into the appropriate space in its parent, then finish the parent, and so forth). now, to make a fully scalable evm, we simply replace the concept of a one-way message with a continuation, and there we go. of course, the question is, do we even want to go this far? first of all, going between substates, such a virtual machine would be incredibly inefficient; if a transaction execution needs to access a total of ten contracts, and each contract is in some random substate, then the process of running through that entire execution will take an average of six blocks per transmission, times two transmissions per sub-call, times ten sub-calls a total of 120 blocks. additionally, we lose synchronicity; if a calls b once and then again, but between the two calls c calls b, then c will have found b in a partially processed state, potentially opening up security holes. finally, it's difficult to combine this mechanism with the concept of reverting transaction execution if transactions run out of gas. thus, it may be easier to not bother with continuations, and rather opt for simple one-way messages; because the language is turing-complete continuations can always be built on top. as a result of the inefficiency and instability of cross-chain messages no matter how they are done, most dapps will want to live entirely inside of a single sub-state, and dapps or contracts that frequently talk to each other will want to live in the same sub-state as well. to prevent absolutely everyone from living on the same sub-state, we can have the gas limits for each substate "spill over" into each other and try to remain similar across substates; then, market forces will naturally ensure that popular substates become more expensive, encouraging marginally indifferent users and dapps to populate fresh new lands. not so fast so, what problems remain? first, there is the data availability problem: what happens when all of the full nodes on a given sub-state disappear? if such a situation happens, the sub-state data disappears forever, and the blockchain will essentially need to be forked from the last block where all of the sub-state data actually is known. this will lead to double-spends, some broken dapps from duplicate messages, etc. hence, we need to essentially be sure that such a thing will never happen. this is a 1-of-n trust model; as long as one honest node stores the data we are fine. single-chain architectures also have this trust model, but the concern increases when the number of nodes expected to store each piece of data decreases as it does here by a factor of 2048. the concern is mitigated by the existence of altruistic nodes including blockchain explorers, but even that will become an issue if the network scales up so much that no single data center will be able to store the entire state. second, there is a fragility problem: if any block anywhere in the system is mis-processed, then that could lead to ripple effects throughout the entire system. a cross-substate message might not be sent, or might be re-sent; coins might be double-spent, and so forth. of course, once a problem is detected it would inevitably be detected, and it could be solved by reverting the whole chain from that point, but it's entirely unclear how often such situations will arise. one fragility solution is to have a separate version of ether in each substate, allowing ethers in different substates to float against each other, and then add message redundancy features to high-level languages, accepting that messages are going to be probabilistic; this would allow the number of nodes verifying each header to shrink to something like 20, allowing even more scalability, though much of that would be absorbed by an increased number of cross-substate messages doing error-correction. a third issue is that the scalability is limited; every transaction needs to be in a substate, and every substate needs to be in a header that every node keeps track of, so if the maximum processing power of a node is n transactions, then the network can process up to n2 transactions. an approach to add further scalability is to make the hypercube structure hierarchical in some fashion imagine the block headers in the header chain as being transactions, and imagine the header chain itself being upgraded from a single-chain model to the exact same hypercube model as described here that would give n3 scalability, and applying it recursively would give something very much like tree chains, with exponential scalability at the cost of increased complexity, and making transactions that go all the way across the state space much more inefficient. finally, fixing the number of substates at 4096 is suboptimal; ideally, the number would grow over time as the state grew. one option is to keep track of the number of transactions per substate, and once the number of transactions per substate exceeds the number of substates we can simply add a dimension to the cube (ie. double the number of substates). more advanced approaches involve using minimal cut algorithms such as the relatively simple karger's algorithm to try to split each substate in half when a dimension is added. however, such approaches are problematic, both because they are complex and because they involve unexpectedly massively increasing the cost and latency of dapps that end up accidentally getting cut across the middle. alternative approaches of course, hypercubing the blockchain is not the only approach to making the blockchain scale. one very promising alternative is to have an ecosystem of multiple blockchains, some application-specific and some ethereum-like generalized scripting environments, and have them "talk to" each other in some fashion in practice, this generally means having all (or at least some) of the blockchains maintain "light clients" of each other inside of their own states. the challenge there is figuring out how to have all of these chains share consensus, particularly in a proof-of-stake context. ideally, all of the chains involved in such a system would reinforce each other, but how would one do that when one can't determine how valuable each coin is? if an attacker has 5% of all a-coins, 3% of all b-coins and 80% of all c-coins, how does a-coin know whether it's b-coin or c-coin that should have the greater weight? one approach is to use what is essentially ripple consensus between chains have each chain decide, either initially on launch or over time via stakeholder consensus, how much it values the consensus input of each other chain, and then allow transitivity effects to ensure that each chain protects every other chain over time. such a system works very well, as it's open to innovation anyone can create new chains at any point with arbitrarily rules, and all the chains can still fit together to reinforce each other; quite likely, in the future we may see such an inter-chain mechanism existing between most chains, and some large chains, perhaps including older ones like bitcoin and architectures like a hypercube-based ethereum 2.0, resting on their own simply for historical reasons. the idea here is for a truly decentralized design: everyone reinforces each other, rather than simply hugging the strongest chain and hoping that that does not fall prey to a black swan attack. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements pyethereum and serpent programming guide | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search pyethereum and serpent programming guide posted by vitalik buterin on april 10, 2014 research & development the content of this tutorial is intended to apply to poc5. most of the instructions given below will not work in the older poc4 implementations of alethzero (c++) and ethereal (go) over the last few weeks, we have made a large number of changes to the ethereum protocol. poc4, introducing a large body of changes made by gavin wood and myself, was announced as an informal description two weeks ago, and has been formally specified in gavin wood’s “yellow paper” at http://gavwood.com/paper.pdf. the protocol spec did change substantially, but at the same time things are solidifying; we know why we want transactions to pay fees instead of contracts, so that’s not likely to change, we know that code and data will be separate, and the byte-based code and memory and 32-byte-block-based stack and storage are unlikely to change, and we know that the workings of the evm in general will be similar to what they are now instead of some kind of elaborate merkle-code-tree construction. poc4 has given myself what i wanted out of ethereum script 2, gavin a much more optimization-friendly vm architecture, and users a shiny new currency. meanwhile, chen houwu, heiko kees and konrad feldmeier have taken the lead as our main python developers, and the networking side of the pyethereum client is getting to the point where it is getting ready to talk to go and c++. at the same time, aside from all of the managerial tasks that are part and parcel of having a key role in a large project, i have taken it upon myself to bring up to speed the pyethereum vm implementation and the compiler for the hll programming language. the purpose of this post will be to provide an in-depth technical tutorial into the workings of pyethereum and serpent, and show you how you can start writing the tools to build your own contracts and applications. the bitcoin expo hackathon is happening today and tomorrow, so feel free to make an ethereum contract your project if you are among those attending. first of all, importantly, hll is no longer called hll; the language is now called serpent. why? because it’s basically python. with recent upgrades to the compiler, serpent is now a highly feature-filled programming language, with powerful features including: arrays (eg. x[0] = 123) array literals (eg. x = [ 34, 56, 78 ]) nested arrays (eg. z = [ 34, [ 5, 6 ], y ]) hex support (eg. receiving_address = 0xb156066c2978d7b9188f2467b815d4c62ae32fe2) string support (eg. x = "cow") inline message calling (eg. usdprice = eth * msg(ethcontract,0,tx.gas-100,[500],1)) out of line message calling (eg. msg(multifeedcontract,0,tx.gas-100,inparray,5,outarray,5)) simple value sending operation (eg. send(receiver, value, tx.gas-100)) returning values (eg. return(45) and return([10,20,30,40],4)) treating message data and storage as arrays (eg. contract.storage[1000] = msg.data[0]) byte arrays (eg. x = bytes(100), setch(x,45,"c")), y = getch(x,45) the intent of the serpent language is to make programming smart contracts and decetralized applications in ethereum as easy as programming boring command line apps is in python. the language is designed to be maximally clean and maximally simple, combining the benefits of a compiled language with an easy-to-use coding experience. just the logic, and nothing but the logic. unfortunately, floating point numbers are missing, as are higher-order constructs like list comprehensions and closures, but aside from that serpent has basically everything that you need. getting started so how do you code in serpent? the first step is to set up the development and execution environment. to do this, first download two libraries: pyethereum and serpent. the simplest way to download is to either download the zip files from github and unpack them, or run git clone http://github.com/ethereum/pyethereum and git clonehttp://github.com/ethereum/serpent. then, enter the pyethereum directory, and run sudo python setup.py install to install pyethereum to your system, and do the same with serpent. now that the software is downloaded, let’s get right to it. to start off, try this:   serpent compile_to_assembly 'x = 5' ["begincode0.endcode0","dup","msize","swap","msize","begincode_0.endcode_0", "dup", "msize", "swap", "msize", "begincode0​.endcode0​","dup","msize","swap","msize","begincode_0", "calldatacopy", "return", "~begincode_0", "#code_begin", 5, 0, "mstore", "#code_end", "~endcode_0"] the compile_to_assembly instruction compiles the code down into an intermediate human-readable “assembly language” format rather than plain old bytecode. using plain old serpent compile would give you the much more incomprehensible but compact 6005515b525b600a37f26005600054. in this case, the “core” of the code is [5, 0, "mstore"], putting the value 5 into memory slot 0, and the rest of the code basically says to return a contract containing that code. another command that you may find useful is serpent get_vars; this will give you a list of all the variables together with their associated memory indices. in this case, you get {'x': 0}, meaning that the compiler is choosing to use the memory index 0 to store the variable x. the last interesting command is parse to convert serpent into an intermediate high-level parse tree. now, since serpent is a programming language, we want to run programs, and so ideally we would like to actually create contracts and run them as quickly as possible. let’s try that. first, open a file, call it “namecoin.se“, and put the following code into it: if !contract.storage[msg.data[0]]: contract.storage[msg.data[0]] = msg.data[1] return(1) else: return(0) this is the two-line namecoin example that we love so much, but embellished with return values to make it easier to work with for this tutorial. typing serpent compile namecoin.se should give:   6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f2 now, let’s see if we can actually get the code running. to do that, the first step is actually to create for ourselves an account. the process here is almost exactly the same as in my python bitcoin library pybitcointools; in general, anyone who is familiar with pybitcointools should feel right at home in pyethereum, although unfortunately in pyethereum it was not really practical to stick to pybitcointools’ “no classes” mantra in the code. the first step is to generate a private key:   pyethtool sha3 cow c85ef7d79691fe79573b1a7064c19c1a9819ebdbd1faaab1a8ec92344438aaf4 in production code, you should obviously replace “cow” with an actually secure password. if you want your account to be a “brainwallet” that you can easily remember, my main advice is to prepend a username, eg. “vbuterin:bl@hbl@hm0nk33y#!$!%”, ensuring that attackers need to target you individually instead of performing a blanket attack on everyone simultaneously; assuming 10000 brainwallet users this reduces your risk from a trial-and-error attack by 99.99%. if you want to use your key later, on any standard linux shell you can also type in key=pyethtool sha3 cow, and then use$key to use the key thereafter. we’ll use that format here from now on, so if you are following along then you should also do both:   key=pyethtool sha3 cow code=serpent compile namecoin.se so now, let’s keep going. addr=pyethtool privtoaddr $key echo $addr cd2a3d9f938e13cd947ec05abc7fe734df8dd826 now, we create a new genesis block, and we'll set the initial endowment to 1018 wei (1 ether) for your address.   genesis=pyethtool mkgenesis $addr 1000000000000000000 echo $genesis f8b2f8aea00000000000000000000000000000000000000000000000000000000000000000a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940000000000000000000000000000000000000000a0bcddd284bf396739c224dba0411566c891c32115feb998a3e2b4e61f3f35582a80834000008087038d7ea4c68000830f4240808080a004994f67dc55b09e814ab7ffc8df3686b4afb2bb53e60eae97ef043fe03fb829c0c0 now that we have that out of the way, we can get to actually doing stuff to the block. the only way to do anything in a blockchain-based architecture, in general, is to create and apply a transaction. here, we will need multiple transactions: the first to create the contract, and then the latter ones to actually use it. here's contract creation:   unsignedtx=pyethtool mkcontract 0 0 $code echo $unsignedtx f83c8085e8d4a510008227108080af6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f2 tx=pyethtool sign $unsignedtx $key echo $tx f87f8085e8d4a510008227108080af6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f21ca04565b5a48b29ef623ad2caffe0917a3fc6a6f1b50f1df06876f3caa6fb4957c6a0123c928257c1f248fb3d362c125a0aea091ab08467efb52f8c3676ca73d727bf or, the easier way:   tx=pyethtool mkcontract 0 0 $code | pyethtool -s sign $key echo $tx f87f8085e8d4a510008227108080af6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f21ca04565b5a48b29ef623ad2caffe0917a3fc6a6f1b50f1df06876f3caa6fb4957c6a0123c928257c1f248fb3d362c125a0aea091ab08467efb52f8c3676ca73d727bf the first field in mkcontract is a nonce, which must be equal to the number of transactions you already sent from that account. the purpose of requiring a nonce is to prevent replay attacks; otherwise, if you sent bob 200 ether, bob could simply replay that transaction over and over again until you run out of money, whereas here due to the nonce requirement the transaction can only go through once. the second field is the amount of ether to send (in the case of contract creation, the amount of ether to initially provide to the contract), and the third field is the code. note that the transaction.contractfunction call also has two more fields between value and recipient: gasprice and startgas. pyethtool is nice to you and initializes these values to 1 szabo (ie. 1012 wei or one millionth of an ether) per gas and 10000 gas, respectively. this will give you a theoretical maximum of 10000 computational steps for the code to run, although in practice it may run out after 1000 if you use many expensive operations. finally, once you create the transaction, you need to sign it with your private key. once that's done, we just, well:   pyethtool applytx genesisgenesis genesistx {"result": "da7ce79725418f4f6e13bf5f520c89cec5f6a974", "block": "f9017ef8d0a00000000000000000000000000000000000000000000000000000000000000000a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940000000000000000000000000000000000000000a00bcec36bf7ffc27418b1746986574526efeb09b34f733039749f291f778d4aaca03575f60ad6c929d7c98a50a12ff1ef9b07ecf3182e74962872064648a66f3da0834000008087038d7ea4c68000830f42408204b08080a004994f67dc55b09e814ab7ffc8df3686b4afb2bb53e60eae97ef043fe03fb829f8a9f8a7b881f87f8085e8d4a510008227108080af6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f21ca04565b5a48b29ef623ad2caffe0917a3fc6a6f1b50f1df06876f3caa6fb4957c6a0123c928257c1f248fb3d362c125a0aea091ab08467efb52f8c3676ca73d727bfa00bcec36bf7ffc27418b1746986574526efeb09b34f733039749f291f778d4aac8204b0c0"} this gives you two values. the first is the address of the contract, and the second is the new block data. note that the block data does not represent the entire block; there is also the state data hidden in the statedb folder. hence, if you try to deserialize the block on a fresh machine it likely will not work. from the values returned, set the first value to contract and the second to med so we can use them later. now, we need to craft a transaction to actually use this contract. suppose we want to register "george" to 45. to do that, however, we first need to do another annoying chore: package up the data. fortunately, the serpent compiler has a utility for doing just that:   data=echo '["george",45]' | serpent -j encode_datalist echo $data 000000000000000000000000000000000000000000000000000067656f726765000000000000000000000000000000000000000000000000000000000000002d the namecoin contract takes data in two fields, the key and the value, so we simply put them into a json array and use serpent to encode it. the encoder can accept strings and numbers as the individual elements in the array. note that unfortunately python's json decoder requires double quotes for internal strings; "['george',45]" would not work. now, we do this:   tx2=pyethtool mktx 1 $contract 0 $data | pyethtool -s sign $key echo $tx2 f8a50185e8d4a5100082271094da7ce79725418f4f6e13bf5f520c89cec5f6a97480b840000000000000000000000000000000000000000000000000000067656f726765000000000000000000000000000000000000000000000000000000000000002d1ba064363844c718f0f38907d39508adb2c2b9134e52e7d436fb20965044c01f41c2a0e1123d26cf810c4ef9d397974e2fc336d16e452d71df3c3d7245b40ed12c603b and:   pyethtool applytx medmed medtx2 {"result": "0000000000000000000000000000000000000000000000000000000000000001", "block": "f9024ef8d0a00000000000000000000000000000000000000000000000000000000000000000a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940000000000000000000000000000000000000000a066d2524d921fadb5056983cf4bb215d339cdaeb7048b8913bfdf8fe867eb5682a0d669d3b5cfb150e4ef7f900cc613b0231abc8551544c389ddcd6668f784c4cb3834000008087038d7ea4c68000830f4240820a8f8080a004994f67dc55b09e814ab7ffc8df3686b4afb2bb53e60eae97ef043fe03fb829f90178f8a7b881f87f8085e8d4a510008227108080af6025515b525b600a37f260003556601b596020356000355760015b525b54602052f260255860005b525b54602052f21ca04565b5a48b29ef623ad2caffe0917a3fc6a6f1b50f1df06876f3caa6fb4957c6a0123c928257c1f248fb3d362c125a0aea091ab08467efb52f8c3676ca73d727bfa00bcec36bf7ffc27418b1746986574526efeb09b34f733039749f291f778d4aac8204b0f8cdb8a7f8a50185e8d4a5100082271094da7ce79725418f4f6e13bf5f520c89cec5f6a97480b840000000000000000000000000000000000000000000000000000067656f726765000000000000000000000000000000000000000000000000000000000000002d1ba064363844c718f0f38907d39508adb2c2b9134e52e7d436fb20965044c01f41c2a0e1123d26cf810c4ef9d397974e2fc336d16e452d71df3c3d7245b40ed12c603ba066d2524d921fadb5056983cf4bb215d339cdaeb7048b8913bfdf8fe867eb5682820a8fc0"} registration successful! the result here is two values, just as before: the first is the new block state, and the second is the response returned by the contract. based on the definition of the contract above, "1" means success. now, just to be sure, let's set end to the block hex returned by the previous command and peek at the state:   pyethtool getstate $end {'nonce': '\x04\x99og\xdcu\xb0\x9e\x81j\xb7\xff\xc8\xdf6\x86\xb4\xaf\xb2\xbbs\xe6\x0e\xae\x97\xef\x04?\xe0?\xb8)', 'min_gas_price': 1000000000000000l, 'extra_data': '', 'state_root': 'f\xd2rm\x92\x1f\xad\xb5\x05i\x83\xcfk\xb2\x15\xd39\xcd\xae\xb7\x04\x8b\x89\x13\xbf\xdf\x8f\xe8g\xebv\x82', 'difficulty': 4194304l, 'timestamp': 0l, 'number': 0l, 'gas_used': 2703l, 'coinbase': '0000000000000000000000000000000000000000', 'tx_list_root': '\xd6i\xd3\xb5\xcf\xb1p\xe4\xef\x7f\x90\x0c\xc6\x13\xb0#\x1a\xbc\x85qtl8\x9d\xdc\xd6f\x8fxll\xb3', 'state': {'0000000000000000000000000000000000000000': {'nonce': 0l, 'balance': 2703000000000000l, 'storage': {}, 'code': ''}, 'da7ce79725418f4f6e13bf5f520c89cec5f6a974': {'nonce': 0l, 'balance': 0l, 'storage': {113685359126373l: 45l}, 'code': '60003556601b596020356000355760015b525b54602052f260255860005b525b54602052f2'}, 'cd2a3d9f938e13cd947ec05abc7fe734df8dd826': {'nonce': 2l, 'balance': 997297000000000000l, 'storage': {}, 'code': ''}}, 'uncles_hash': '\x1d\xccm\xe8\xde\xc7]z\xab\x85\xb5g\xb6\xcc\xd4\x1a\xd3\x12e\x1b\x94\x8at\x13\xf0\xa1b\xfd@\xd4\x93g', 'prevhash': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'gas_limit': 1000000l} you can see the contract account near the beginning of the state description, with "george" registered to 45 as expected. we're done! as an exercise, try constructing two more transactions, one registering "george" to 60 and another registering "harry" to 80. if you apply them all sequentially after these two, the one registering "george" to 60 should return 0, but the one registering "harry" to 80 should succceed. doing it in python that's pyethtool, the command line utility. now, how does it work using pyethereum itself? as it turns out, it's surprisingly easy. here's the session:   >>> import serpent >>> from pyethereum import transactions, blocks, processblock, utils >>> code = serpent.compile(open('namecoin.se').read()) >>> key = utils.sha3('cow') >>> addr = utils.privtoaddr(key) >>> genesis = blocks.genesis({ addr: 10**18 }) >>> tx1 = transactions.contract(0,10**12,10000,0,code).sign(key) >>> result, contract = processblock.apply_tx(genesis,tx1) >>> tx2 = transactions.transaction(1,10**12,10000,contract,0,serpent.encode_datalist(['george',45])).sign(key) >>> result, ans = processblock.apply_tx(genesis,tx2) >>> serpent.decode_datalist(ans) [1] >>> genesis.to_dict() 'nonce': '\x04\x99og\xdcu\xb0\x9e\x81j\xb7\xff\xc8\xdf6\x86\xb4\xaf\xb2\xbbs\xe6\x0e\xae\x97\xef\x04?\xe0?\xb8)', 'min_gas_price': 1000000000000000l, 'extra_data': '', 'state_root': '', 'difficulty': 4194304, 'timestamp': 0, 'number': 0, 'gas_used': 2712l, 'coinbase': '0000000000000000000000000000000000000000', 'tx_list_root': '\x17\x90\x87\x966\xbdb!\x14|r\xb0& \xb04\x90\xb9bs\x12\x85\x90\xdab\xed\x83n*\x8ee\x8e', 'state': {'0000000000000000000000000000000000000000': {'nonce': 0l, 'balance': 2712000000000000l, 'storage': {}, 'code': ''}, 'da7ce79725418f4f6e13bf5f520c89cec5f6a974': {'nonce': 0l, 'balance': 0l, 'storage': {113685359126373l: 45l}, 'code': '60003556601e596020356000355760015b525b54602052f260285860005b525b54602052f2'}, 'cd2a3d9f938e13cd947ec05abc7fe734df8dd826': {'nonce': 2l, 'balance': 997288000000000000l, 'storage': {}, 'code': ''}}, 'uncles_hash': '\x1d\xccm\xe8\xde\xc7]z\xab\x85\xb5g\xb6\xcc\xd4\x1a\xd3\x12e\x1b\x94\x8at\x13\xf0\xa1b\xfd@\xd4\x93g', 'prevhash': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'gas_limit': 1000000} >>> genesis.get_balance(addr) 997288000000000000l >>> genesis.get_storage_data(contract,'george') 45l another important command is processblock.debug = 1; this starts printing code execution step by step, helping you debug what is wrong in your contract code or my pyethereum vm or serpent implementation! getting into the code so that's your introduction to how to use pyethereum. now, let's get into the most fun part, writing contracts. for reading efficiency, let's provide the namecoin contract again: if !contract.storage[msg.data[0]]: contract.storage[msg.data[0]] = msg.data[1] return(1) else: return(0) what does this contract do? essentially, this contract implements a name registration database by simply using that as the sole function of the long-term storage of the contract. contract code theoretically has three places to put data: stack, memory and storage. of those three, stack and memory are used implicitly in serpent to support arithmetic and variables, but long-term storage is the only one that survives once execution is over. here, when you register "george" to 45, the contract first checks ifcontract.storage["george"] is not nonzero, ie. is zero. if it is, then it sets that storage index to the value provided, 45, and then returns 1. if it is not, then it returns zero. note that this contract has no way for other contracts to access it; it is only really usable by external applications. more advanced name registries would have an api for contracts to fetch the data associated with a name as well. now, on to a more intricate example: init: contract.storage[0xcd2a3d9f938e13cd947ec05abc7fe734df8dd826] = 1000000 code: if msg.datasize == 1: addr = msg.data[0] return(contract.storage[addr]) else: from = msg.sender fromvalue = contract.storage[from] to = msg.data[0] value = msg.data[1] if fromvalue >= value: contract.storage[from] = fromvalue value contract.storage[to] = contract.storage[to] + value return(1) else: return(0) this is the "currency contract", or more precisely an embellished version of it with return values to make debugging easier. this contract is interesting for several reasons. first, it has an initialization step, which gets called when the contract is first made. this initializes an account with 1000000 currency units owned by that account. after that, there are two code paths. first, incoming messages might contain only one data field. in that case, these messages are treated as balance queries, and simply return the balance of the queried address. note that msg.data[0] provides the integer at bytes 0...31 of the transaction data, msg.data[1] provides the integer at bytes 32...63, and so forth. this is a convenience introduced in serpent; the underlying transaction data is all byte-based. incidentally, this is why we needed to use serpent's encode_datalist function to generate the transaction data. second, incoming messages might contain two data fields. in that case, the messages are treated as requests to send to that address. the sender is inferred from the sender of the message, and the recipient and the value are taken from the first two fields (ie. first 64 bytes) in msg.data. if there is enough money to transfer, it transfers the money and returns 1; otherwise it returns 0. challenge: create a currency contract which takes a fee, denominated in its internal currency, from every transaction, and refunds a small amount of ether to everyone sending a successful transaction, so people (or contracts) who want to deal in this currency would not have to worry about simultaneously maintaining currency and ether balances themselves. the contract would also include a third transaction type, perhaps taking 0 arguments, through which someone can buy internal currency units from the contract by sending it ether. the contract should keep track of two variables: its own balance in its currency, and its ether balance, and it should dynamically adjust the transaction fee and the exchange rate in order to keep both its ether balance and its internal currency balance in baluh, in an approximate equilibrium. contracts calling contracts this is a proprietary data feed contract: owner = 0xcd2a3d9f938e13cd947ec05abc7fe734df8dd826 if msg.sender == owner and msg.datasize == 2: contract.storage[msg.data[0]] = msg.data[1] return(1) else: return(contract.storage[msg.data[0]]) this contract is designed to work as a key/value that can be edited only by its owner, but also also allows anyone to query its contents; the point is for the owner to use various storage indices to record changing data like the usd price of ether. here, there are two main "clauses" in the contract, one for modifying storage which triggers if a key and a value are provided and the message originates from the contract's owner, and the other for just reading storage. the msg.datasize variable tells you the number of 32-byte data fields there is in the message data. there are no particularly new features here; this contract is actually fairly simple, and i encourage you to first follow and make sure you understand the logic involved and then play with the contract, instantiating it in a block and then pushing set and query transactions to it. the interesting part, however, comes when we use this contract inside of another contract. meet this monstrosity, a hedging contract: if !contract.storage[1000]: contract.storage[1000] = msg.sender contract.storage[1002] = msg.value contract.storage[1003] = msg.data[0] contract.storage[1004] = msg.data[1] return(1) elif !contract.storage[1001]: ethvalue = contract.storage[1002] if msg.value >= ethvalue: contract.storage[1001] = msg.sender datasource = contract.storage[1003] dataindex = contract.storage[1004] othervalue = ethvalue * msg(datasource,0,tx.gas-100,[dataindex],1) contract.storage[1005] = othervalue contract.storage[1006] = block.timestamp + 86400 return([2,othervalue],2) else: datasource = contract.storage[1003] dataindex = contract.storage[1004] othervalue = contract.storage[1005] ethvalue = othervalue / msg(dataindex,0,tx.gas-100,[datasource],1) if ethvalue >= contract.balance: send(contract.storage[1000],contract.balance,tx.gas-100) return(3) elif block.timestamp > contract.storage[1006]: send(contract.storage[1001],contract.balance ethvalue,tx.gas-100) send(contract.storage[1000],ethvalue,tx.gas-100) return(4) else: return(5) this contract is bulky because it's designed to be more testing-friendly; an optimal implementation is roughly half the size. the contract works as follows: party a sends in x ether alongside a data feed contract d and a currency code c as data items, and is registered at contract storage index 1000. x, d and c are registered in contract storage indices 1002, 1003 and 1004. in this case, suppose that the currency code represents usd. party b sends in x ether, and is registered at contract storage index 1001. the contract then calls d with data c to determine the price of ether in the given currency, and uses this to compute v, the amount of value in usd sent by each party. v is stored at index 1005, and an expiry time set to 24 hours in the future is stored at index 1006. maybe, the price of ether in usd drops by more than 50%. if this happens, then there is not enough ether in the contract altogether to pay v usd. to prevent this, as soon as the price slips under the 50% mark, anyone (usually a) can ping the contract to withdraw all 2x ether into a's address and thereby recover to a's address almost all of the amount, as measured in usd, that a put in, and leave b with nothing. if this happens, the contract returns 3. otherwise, after one day, anyone can send a transaction to "ping" the contract and cause it to send v usd worth of ether to a and the remaining ether to b, returning 4. if there is no "margin call" or "expiry" event, then a ping to the contract does nothing and returns 5. the point of the hedging contract is that a benefits by always getting back the same quantity of usd that he put in, and b benefits if he believes that the value of ether will go up, since a 10% rise in the ether price will, in this circumstance, give him a 20% profit. usd can of course be substituted with anything, including cny, gold or the consumer price index. the important new features explored here are msg, send and array literals. msg and send are both ways of sending message to other contracts. the syntaxes are:   send(to, value, gas) out = msg(to¸ value, gas, datastart, datalength) msg(to, value, gas, datastart, datalength, outstart, outlength) send is simpler, assuming that all you want to do is send money with no bells and whistles involved. the latter two are equivalent ways of sending a message to another contract, differing only in how they handle the output: the first caps output to 32 bytes and sticks it straight into a variable, whereas the second takes in two arguments for the position in memory where to dump the output. the "output" of a message is blank if the recipient is not-yet-existent, an externally owned account, or does not explicitly specify a return value, and if the output does specify a return value then the output is that value ("value" in this context being an arbitrary-length byte array, not a 32-byte number). these two are thus both ways of saying the same thing: d = array(3) d[0] = 5 d[1] = 10 d[2] = 15 x = msg(a, b, c, d, 3) and: d = array(3) d[0] = 5 d[1] = 10 d[2] = 15 w = array(1) msg(a, b, c, d, 3, w, 1) x = w[0] in the contract example above, we used the data feed contract to provide the price of ether in usd, and then directly plugged it into the formula othervalue = ethvalue * msg(datasource,0,tx.gas-100,[dataindex],1). array literals are another nice convenience feature; the truly optimal way to write the above code is as follows: x = msg(a, b, c, [5, 10, 15], 3) note that you unfortunately still need to specify the array length. however, here the array itself is created and referenced all inline, without needing to manually set things up. all of the magic is done by the serpent compiler. so that's basically it for today. what might you want to code in serpent? well, here are a few possibilities: schellingcoin a contract-based implementation of justdice. some skeleton code for a decentralized organization. a board game (eg. chess, go) a decentralized exchange, with a contract-based order book, between ether and the sub-currency contract given above. any of the other examples in our whitepaper enjoy, and have fun! also, if you do find any bugs in pyethereum or serpent, please be sure to point them out. see also: list of serpent language operations previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle layer 1 should be innovative in the short term but less in the long term 2018 aug 26 see all posts see update 2018-08-29 one of the key tradeoffs in blockchain design is whether to build more functionality into base-layer blockchains themselves ("layer 1"), or to build it into protocols that live on top of the blockchain, and can be created and modified without changing the blockchain itself ("layer 2"). the tradeoff has so far shown itself most in the scaling debates, with block size increases (and sharding) on one side and layer-2 solutions like plasma and channels on the other, and to some extent blockchain governance, with loss and theft recovery being solvable by either the dao fork or generalizations thereof such as eip 867, or by layer-2 solutions such as reversible ether (reth). so which approach is ultimately better? those who know me well, or have seen me out myself as a dirty centrist, know that i will inevitably say "some of both". however, in the longer term, i do think that as blockchains become more and more mature, layer 1 will necessarily stabilize, and layer 2 will take on more and more of the burden of ongoing innovation and change. there are several reasons why. the first is that layer 1 solutions require ongoing protocol change to happen at the base protocol layer, base layer protocol change requires governance, and it has still not been shown that, in the long term, highly "activist" blockchain governance can continue without causing ongoing political uncertainty or collapsing into centralization. to take an example from another sphere, consider moxie marlinspike's defense of signal's centralized and non-federated nature. a document by a company defending its right to maintain control over an ecosystem it depends on for its key business should of course be viewed with massive grains of salt, but one can still benefit from the arguments. quoting: one of the controversial things we did with signal early on was to build it as an unfederated service. nothing about any of the protocols we've developed requires centralization; it's entirely possible to build a federated signal protocol-based messenger, but i no longer believe that it is possible to build a competitive federated messenger at all. and: their retort was "that's dumb, how far would the internet have gotten without interoperable protocols defined by 3rd parties?" i thought about it. we got to the first production version of ip, and have been trying for the past 20 years to switch to a second production version of ip with limited success. we got to http version 1.1 in 1997, and have been stuck there until now. likewise, smtp, irc, dns, xmpp, are all similarly frozen in time circa the late 1990s. to answer his question, that's how far the internet got. it got to the late 90s. that has taken us pretty far, but it's undeniable that once you federate your protocol, it becomes very difficult to make changes. and right now, at the application level, things that stand still don't fare very well in a world where the ecosystem is moving ... so long as federation means stasis while centralization means movement, federated protocols are going to have trouble existing in a software climate that demands movement as it does today. at this point in time, and in the medium term going forward, it seems clear that decentralized application platforms, cryptocurrency payments, identity systems, reputation systems, decentralized exchange mechanisms, auctions, privacy solutions, programming languages that support privacy solutions, and most other interesting things that can be done on blockchains are spheres where there will continue to be significant and ongoing innovation. decentralized application platforms often need continued reductions in confirmation time, payments need fast confirmations, low transaction costs, privacy, and many other built-in features, exchanges are appearing in many shapes and sizes including on-chain automated market makers, frequent batch auctions, combinatorial auctions and more. hence, "building in" any of these into a base layer blockchain would be a bad idea, as it would create a high level of governance overhead as the platform would have to continually discuss, implement and coordinate newly discovered technical improvements. for the same reason federated messengers have a hard time getting off the ground without re-centralizing, blockchains would also need to choose between adopting activist governance, with the perils that entails, and falling behind newly appearing alternatives. even ethereum's limited level of application-specific functionality, precompiles, has seen some of this effect. less than a year ago, ethereum adopted the byzantium hard fork, including operations to facilitate elliptic curve operations needed for ring signatures, zk-snarks and other applications, using the alt-bn128 curve. now, zcash and other blockchains are moving toward bls-12-381, and ethereum would need to fork again to catch up. in part to avoid having similar problems in the future, the ethereum community is looking to upgrade the evm to e-wasm, a virtual machine that is sufficiently more efficient that there is far less need to incorporate application-specific precompiles. but there is also a second argument in favor of layer 2 solutions, one that does not depend on speed of anticipated technical development: sometimes there are inevitable tradeoffs, with no single globally optimal solution. this is less easily visible in ethereum 1.0-style blockchains, where there are certain models that are reasonably universal (eg. ethereum's account-based model is one). in sharded blockchains, however, one type of question that does not exist in ethereum today crops up: how to do cross-shard transactions? that is, suppose that the blockchain state has regions a and b, where few or no nodes are processing both a and b. how does the system handle transactions that affect both a and b? the current answer involves asynchronous cross-shard communication, which is sufficient for transferring assets and some other applications, but insufficient for many others. synchronous operations (eg. to solve the train and hotel problem) can be bolted on top with cross-shard yanking, but this requires multiple rounds of cross-shard interaction, leading to significant delays. we can solve these problems with a synchronous execution scheme, but this comes with several tradeoffs: the system cannot process more than one transaction for the same account per block transactions must declare in advance what shards and addresses they affect there is a high risk of any given transaction failing (and still being required to pay fees!) if the transaction is only accepted in some of the shards that it affects but not others it seems very likely that a better scheme can be developed, but it would be more complex, and may well have limitations that this scheme does not. there are known results preventing perfection; at the very least, amdahl's law puts a hard limit on the ability of some applications and some types of interaction to process more transactions per second through parallelization. so how do we create an environment where better schemes can be tested and deployed? the answer is an idea that can be credited to justin drake: layer 2 execution engines. users would be able to send assets into a "bridge contract", which would calculate (using some indirect technique such as interactive verification or zk-snarks) state roots using some alternative set of rules for processing the blockchain (think of this as equivalent to layer-two "meta-protocols" like mastercoin/omni and counterparty on top of bitcoin, except because of the bridge contract these protocols would be able to handle assets whose "base ledger" is defined on the underlying protocol), and which would process withdrawals if and only if the alternative ruleset generates a withdrawal request. note that anyone can create a layer 2 execution engine at any time, different users can use different execution engines, and one can switch from one execution engine to any other, or to the base protocol, fairly quickly. the base blockchain no longer has to worry about being an optimal smart contract processing engine; it need only be a data availability layer with execution rules that are quasi-turing-complete so that any layer 2 bridge contract can be built on top, and that allow basic operations to carry state between shards (in fact, only eth transfers being fungible across shards is sufficient, but it takes very little effort to also allow cross-shard calls, so we may as well support them), but does not require complexity beyond that. note also that layer 2 execution engines can have different state management rules than layer 1, eg. not having storage rent; anything goes, as it's the responsibility of the users of that specific execution engine to make sure that it is sustainable, and if they fail to do so the consequences are contained to within the users of that particular execution engine. in the long run, layer 1 would not be actively competing on all of these improvements; it would simply provide a stable platform for the layer 2 innovation to happen on top. does this mean that, say, sharding is a bad idea, and we should keep the blockchain size and state small so that even 10 year old computers can process everyone's transactions? absolutely not. even if execution engines are something that gets partially or fully moved to layer 2, consensus on data ordering and availability is still a highly generalizable and necessary function; to see how difficult layer 2 execution engines are without layer 1 scalable data availability consensus, see the difficulties in plasma research, and its difficulty of naturally extending to fully general purpose blockchains, for an example. and if people want to throw a hundred megabytes per second of data into a system where they need consensus on availability, then we need a hundred megabytes per second of data availability consensus. additionally, layer 1 can still improve on reducing latency; if layer 1 is slow, the only strategy for achieving very low latency is state channels, which often have high capital requirements and can be difficult to generalize. state channels will always beat layer 1 blockchains in latency as state channels require only a single network message, but in those cases where state channels do not work well, layer 1 blockchains can still come closer than they do today. hence, the other extreme position, that blockchain base layers can be truly absolutely minimal, and not bother with either a quasi-turing-complete execution engine or scalability to beyond the capacity of a single node, is also clearly false; there is a certain minimal level of complexity that is required for base layers to be powerful enough for applications to build on top of them, and we have not yet reached that level. additional complexity is needed, though it should be chosen very carefully to make sure that it is maximally general purpose, and not targeted toward specific applications or technologies that will go out of fashion in two years due to loss of interest or better alternatives. and even in the future base layers will need to continue to make some upgrades, especially if new technologies (eg. starks reaching higher levels of maturity) allow them to achieve stronger properties than they could before, though developers today can take care to make base layer platforms maximally forward-compatible with such potential improvements. so it will continue to be true that a balance between layer 1 and layer 2 improvements is needed to continue improving scalability, privacy and versatility, though layer 2 will continue to take up a larger and larger share of the innovation over time. update 2018.08.29: justin drake pointed out to me another good reason why some features may be best implemented on layer 1: those features are public goods, and so could not be efficiently or reliably funded with feature-specific use fees, and hence are best paid for by subsidies paid out of issuance or burned transaction fees. one possible example of this is secure random number generation, and another is generation of zero knowledge proofs for more efficient client validation of correctness of various claims about blockchain contents or state. dark mode toggle a quick gasprice market analysis 2017 dec 14 see all posts here is a file that contains data, extracted from geth, about transaction fees in every block between 4710000 and 4730000. for each block, it contains an object of the form: { "block":4710000, "coinbase":"0x829bd824b016326a401d083b33d092293333a830", "deciles":[40,40.1,44.100030001,44.100030001,44.100030001,44.100030001,44.100030001,44.100030001,50,66.150044,100] ,"free":10248, "timedelta":8 } the "deciles" variable contains 11 values, where the lowest is the lowest gasprice in each block, the next is the gasprice that only 10% of other transaction gasprices are lower than, and so forth; the last is the highest gasprice in each block. this gives us a convenient summary of the distribution of transaction fees that each block contains. we can use this data to perform some interesting analyses. first, a chart of the deciles, taking 50-block moving averages to smooth it out: what we see is a gasprice market that seems to actually stay reasonably stable over the course of more than three days. there are a few occasional spikes, most notably the one around block 4720000, but otherwise the deciles all stay within the same band all the way through. the only exception is the highest gasprice transaction (that red squiggle at the top left), which fluctuates wildly because it can be pushed upward by a single very-high-gasprice transaction. we can try to interpret the data in another way: by calculating, for each gasprice level, the average number of blocks that you need to wait until you see a block where the lowest gasprice included is lower than that gasprice. assuming that miners are rational and all have the same view (implying that if the lowest gasprice in a block is x, then that means there are no more transactions with gasprices above x waiting to be included), this might be a good proxy for the average amount of time that a transaction sender needs to wait to get included if they use that gasprice. the stats are: there is clear clustering going on at the 4, 10 and 20 levels; it seems to be an underexploited strategy to send transactions with fees slightly above these levels, getting in before the crowd of transactions right at the level but only paying a little more. however, there is quite a bit of evidence that miners do not have the same view; that is, some miners see a very different set of transactions from other miners. first of all, we can filter blocks by miner address, and check what the deciles of each miner are. here is the output of this data, splitting by 2000-block ranges so we can spot behavior that is consistent across the entire period, and filtering out miners that mine less than 10 blocks in any period, as well as filtering out blocks with more 21000 free gas (high levels of free gas may signify an abnormally high minimum gas price policy, like for example 0x6a7a43be33ba930fe58f34e07d0ad6ba7adb9b1f at ~40 gwei and 0xb75d1e62b10e4ba91315c4aa3facc536f8a922f5 at ~10 gwei). we get: 0x829bd824b016326a401d083b33d092293333a830 [30, 28, 27, 21, 28, 34, 23, 24, 32, 32] 0xea674fdde714fd979de3edf0f56aa9716b898ec8 [17, 11, 10, 15, 17, 23, 17, 13, 16, 17] 0x5a0b54d5dc17e0aadc383d2db43b0a0d3e029c4c [31, 17, 20, 18, 16, 27, 21, 15, 21, 21] 0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5 [20, 16, 19, 14, 17, 18, 17, 14, 15, 15] 0xb2930b35844a230f00e51431acae96fe543a0347 [21, 17, 19, 17, 17, 25, 17, 16, 19, 19] 0x180ba8f73897c0cb26d76265fc7868cfd936e617 [13, 13, 15, 18, 12, 26, 16, 13, 20, 20] 0xf3b9d2c81f2b24b0fa0acaaa865b7d9ced5fc2fb [26, 25, 23, 21, 22, 28, 25, 24, 26, 25] 0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01 [17, 21, 17, 14, 21, 32, 14, 14, 19, 23] 0x2a65aca4d5fc5b5c859090a6c34d164135398226 [26, 24, 20, 16, 22, 33, 20, 18, 24, 24] the first miner is consistently higher than the others; the last is also higher than average, and the second is consistently among the lowest. another thing we can look at is timestamp differences the difference between a block's timestamp and its parent. there is a clear correlation between timestamp difference and lowest gasprice: this makes a lot of sense, as a block that comes right after another block should be cleaning up only the transactions that are too low in gasprice for the parent block to have included, and a block that comes a long time after its predecessor would have many more not-yet-included transactions to choose from. the differences are large, suggesting that a single block is enough to bite off a very substantial chunk of the unconfirmed transaction pool, adding to the evidence that most transactions are included quite quickly. however, if we look at the data in more detail, we see very many instances of blocks with low timestamp differences that contain many transactions with higher gasprices than their parents. sometimes we do see blocks that actually look like they clean up what their parents could not, like this: {"block":4710093,"coinbase":"0x5a0b54d5dc17e0aadc383d2db43b0a0d3e029c4c","deciles":[25,40,40,40,40,40,40,43,50,64.100030001,120],"free":6030,"timedelta":8}, {"block":4710094,"coinbase":"0xea674fdde714fd979de3edf0f56aa9716b898ec8","deciles":[4,16,20,20,21,21,22,29,30,40,59],"free":963366,"timedelta":2}, but sometimes we see this: {"block":4710372,"coinbase":"0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5","deciles":[1,30,35,40,40,40,40,40,40,55,100],"free":13320,"timedelta":7}, {"block":4710373,"coinbase":"0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5","deciles":[1,32,32,40,40,56,56,56,56,70,80],"free":1672720,"timedelta":2} and sometimes we see miners suddenly including many 1-gwei transactions: {"block":4710379,"coinbase":"0x5a0b54d5dc17e0aadc383d2db43b0a0d3e029c4c","deciles":[21,25,31,40,40,40,40,40,40,50,80],"free":4979,"timedelta":13}, {"block":4710380,"coinbase":"0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5","deciles":[1,1,1,1,1,1,40,45,55,61.10006,2067.909560115],"free":16730,"timedelta":35} this strongly suggests that a miner including transactions with gasprice x should not be taken as evidence that there are not still many transactions with gasprice higher than x left to process. this is likely because of imperfections in network propagation. in general, however, what we see seems to be a rather well-functioning fee market, though there is still room to improve in fee estimation and, most importantly of all, continuing to work hard to improve base-chain scalability so that more transactions can get included in the first place. dark mode toggle control as liability 2019 may 09 see all posts the regulatory and legal environment around internet-based services and applications has changed considerably over the last decade. when large-scale social networking platforms first became popular in the 2000s, the general attitude toward mass data collection was essentially "why not?". this was the age of mark zuckerberg saying the age of privacy is over and eric schmidt arguing, "if you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place." and it made personal sense for them to argue this: every bit of data you can get about others was a potential machine learning advantage for you, every single restriction a weakness, and if something happened to that data, the costs were relatively minor. ten years later, things are very different. it is especially worth zooming in on a few particular trends. privacy. over the last ten years, a number of privacy laws have been passed, most aggressively in europe but also elsewhere, but the most recent is the gdpr. the gdpr has many parts, but among the most prominent are: (i) requirements for explicit consent, (ii) requirement to have a legal basis to process data, (iii) users' right to download all their data, (iv) users' right to require you to delete all their data. other jurisdictions are exploring similar rules. data localization rules. india, russia and many other jurisdictions increasingly have or are exploring rules that require data on users within the country to be stored inside the country. and even when explicit laws do not exist, there's a growing shift toward concern (eg. 1 2) around data being moved to countries that are perceived to not sufficiently protect it. sharing economy regulation. sharing economy companies such as uber are having a hard time arguing to courts that, given the extent to which their applications control and direct drivers' activity, they should not be legally classified as employers. cryptocurrency regulation. a recent fincen guidance attempts to clarify what categories of cryptocurrency-related activity are and are not subject to regulatory licensing requirements in the united states. running a hosted wallet? regulated. running a wallet where the user controls their funds? not regulated. running an anonymizing mixing service? if you're running it, regulated. if you're just writing code... not regulated. as emin gun sirer points out, the fincen cryptocurrency guidance is not at all haphazard; rather, it's trying to separate out categories of applications where the developer is actively controlling funds, from applications where the developer has no control. the guidance carefully separates out how multisignature wallets, where keys are held both by the operator and the user, are sometimes regulated and sometimes not: if the multiple-signature wallet provider restricts its role to creating un-hosted wallets that require adding a second authorization key to the wallet owner's private key in order to validate and complete transactions, the provider is not a money transmitter because it does not accept and transmit value. on the other hand, if ... the value is represented as an entry in the accounts of the provider, the owner does not interact with the payment system directly, or the provider maintains total independent control of the value, the provider will also qualify as a money transmitter. although these events are taking place across a variety of contexts and industries, i would argue that there is a common trend at play. and the trend is this: control over users' data and digital possessions and activity is rapidly moving from an asset to a liability. before, every bit of control you have was good: it gives you more flexibility to earn revenue, if not now then in the future. now, every bit of control you have is a liability: you might be regulated because of it. if you exhibit control over your users' cryptocurrency, you are a money transmitter. if you have "sole discretion over fares, and can charge drivers a cancellation fee if they choose not to take a ride, prohibit drivers from picking up passengers not using the app and suspend or deactivate drivers' accounts", you are an employer. if you control your users' data, you're required to make sure you can argue just cause, have a compliance officer, and give your users access to download or delete the data. if you are an application builder, and you are both lazy and fear legal trouble, there is one easy way to make sure that you violate none of the above new rules: don't build applications that centralize control. if you build a wallet where the user holds their private keys, you really are still "just a software provider". if you build a "decentralized uber" that really is just a slick ui combining a payment system, a reputation system and a search engine, and don't control the components yourself, you really won't get hit by many of the same legal issues. if you build a website that just... doesn't collect data (static web pages? but that's impossible!) you don't have to even think about the gdpr. this kind of approach is of course not realistic for everyone. there will continue to be many cases where going without the conveniences of centralized control simply sacrifices too much for both developers and users, and there are also cases where the business model considerations mandate a more centralized approach (eg. it's easier to prevent non-paying users from using software if the software stays on your servers) win out. but we're definitely very far from having explored the full range of possibilities that more decentralized approaches offer. generally, unintended consequences of laws, discouraging entire categories of activity when one wanted to only surgically forbid a few specific things, are considered to be a bad thing. here though, i would argue that the forced shift in developers' mindsets, from "i want to control more things just in case" to "i want to control fewer things just in case", also has many positive consequences. voluntarily giving up control, and voluntarily taking steps to deprive oneself of the ability to do mischief, does not come naturally to many people, and while ideologically-driven decentralization-maximizing projects exist today, it's not at all obvious at first glance that such services will continue to dominate as the industry mainstreams. what this trend in regulation does, however, is that it gives a big nudge in favor of those applications that are willing to take the centralization-minimizing, user-sovereignty-maximizing "can't be evil" route. hence, even though these regulatory changes are arguably not pro-freedom, at least if one is concerned with the freedom of application developers, and the transformation of the internet into a subject of political focus is bound to have many negative knock-on effects, the particular trend of control becoming a liability is in a strange way even more pro-cypherpunk (even if not intentionally!) than policies of maximizing total freedom for application developers would have been. though the present-day regulatory landscape is very far from an optimal one from the point of view of almost anyone's preferences, it has unintentionally dealt the movement for minimizing unneeded centralization and maximizing users' control of their own assets, private keys and data a surprisingly strong hand to execute on its vision. and it would be highly beneficial to the movement to take advantage of it. dark mode toggle on collusion 2019 apr 03 see all posts special thanks to glen weyl, phil daian and jinglan wang for review over the last few years there has been an increasing interest in using deliberately engineered economic incentives and mechanism design to align behavior of participants in various contexts. in the blockchain space, mechanism design first and foremost provides the security for the blockchain itself, encouraging miners or proof of stake validators to participate honestly, but more recently it is being applied in prediction markets, "token curated registries" and many other contexts. the nascent radicalxchange movement has meanwhile spawned experimentation with harberger taxes, quadratic voting, quadratic financing and more. more recently, there has also been growing interest in using token-based incentives to try to encourage quality posts in social media. however, as development of these systems moves closer from theory to practice, there are a number of challenges that need to be addressed, challenges that i would argue have not yet been adequately confronted. as a recent example of this move from theory toward deployment, bihu, a chinese platform that has recently released a coin-based mechanism for encouraging people to write posts. the basic mechanism (see whitepaper in chinese here) is that if a user of the platform holds key tokens, they have the ability to stake those key tokens on articles; every user can make \(k\) "upvotes" per day, and the "weight" of each upvote is proportional to the stake of the user making the upvote. articles with a greater quantity of stake upvoting them appear more prominently, and the author of an article gets a reward of key tokens roughly proportional to the quantity of key upvoting that article. this is an oversimplification and the actual mechanism has some nonlinearities baked into it, but they are not essential to the basic functioning of the mechanism. key has value because it can be used in various ways inside the platform, but particularly a percentage of all ad revenues get used to buy and burn key (yay, big thumbs up to them for doing this and not making yet another medium of exchange token!). this kind of design is far from unique; incentivizing online content creation is something that very many people care about, and there have been many designs of a similar character, as well as some fairly different designs. and in this case this particular platform is already being used significantly: a few months ago, the ethereum trading subreddit /r/ethtrader introduced a somewhat similar experimental feature where a token called "donuts" is issued to users that make comments that get upvoted, with a set amount of donuts issued weekly to users in proportion to how many upvotes their comments received. the donuts could be used to buy the right to set the contents of the banner at the top of the subreddit, and could also be used to vote in community polls. however, unlike what happens in the key system, here the reward that b receives when a upvotes b is not proportional to a's existing coin supply; instead, each reddit account has an equal ability to contribute to other reddit accounts. these kinds of experiments, attempting to reward quality content creation in a way that goes beyond the known limitations of donations/microtipping, are very valuable; under-compensation of user-generated internet content is a very significant problem in society in general (see "liberal radicalism" and "data as labor"), and it's heartening to see crypto communities attempting to use the power of mechanism design to make inroads on solving it. but unfortunately, these systems are also vulnerable to attack. self-voting, plutocracy and bribes here is how one might economically attack the design proposed above. suppose that some wealthy user acquires some quantity \(n\) of tokens, and as a result each of the user's \(k\) upvotes gives the recipient a reward of \(n \cdot q\) (\(q\) here probably being a very small number, eg. think \(q = 0.000001\)). the user simply upvotes their own sockpuppet accounts, giving themselves the reward of \(n \cdot k \cdot q\). then, the system simply collapses into each user having an "interest rate" of \(k \cdot q\) per period, and the mechanism accomplishes nothing else. the actual bihu mechanism seemed to anticipate this, and has some superlinear logic where articles with more key upvoting them gain a disproportionately greater reward, seemingly to encourage upvoting popular posts rather than self-upvoting. it's a common pattern among coin voting governance systems to add this kind of superlinearity to prevent self-voting from undermining the entire system; most dpos schemes have a limited number of delegate slots with zero rewards for anyone who does not get enough votes to join one of the slots, with similar effect. but these schemes invariably introduce two new weaknesses: they subsidize plutocracy, as very wealthy individuals and cartels can still get enough funds to self-upvote. they can be circumvented by users bribing other users to vote for them en masse. bribing attacks may sound farfetched (who here has ever accepted a bribe in real life?), but in a mature ecosystem they are much more realistic than they seem. in most contexts where bribing has taken place in the blockchain space, the operators use a euphemistic new name to give the concept a friendly face: it's not a bribe, it's a "staking pool" that "shares dividends". bribes can even be obfuscated: imagine a cryptocurrency exchange that offers zero fees and spends the effort to make an abnormally good user interface, and does not even try to collect a profit; instead, it uses coins that users deposit to participate in various coin voting systems. there will also inevitably be people that see in-group collusion as just plain normal; see a recent scandal involving eos dpos for one example: finally, there is the possibility of a "negative bribe", ie. blackmail or coercion, threatening participants with harm unless they act inside the mechanism in a certain way. in the /r/ethtrader experiment, fear of people coming in and buying donuts to shift governance polls led to the community deciding to make only locked (ie. untradeable) donuts eligible for use in voting. but there's an even cheaper attack than buying donuts (an attack that can be thought of as a kind of obfuscated bribe): renting them. if an attacker is already holding eth, they can use it as collateral on a platform like compound to take out a loan of some token, giving you the full right to use that token for whatever purpose including participating in votes, and when they're done they simply send the tokens back to the loan contract to get their collateral back all without having to endure even a second of price exposure to the token that they just used to swing a coin vote, even if the coin vote mechanism includes a time lockup (as eg. bihu does). in every case, issues around bribing, and accidentally over-empowering well-connected and wealthy participants, prove surprisingly difficult to avoid. identity some systems attempt to mitigate the plutocratic aspects of coin voting by making use of an identity system. in the case of the /r/ethtrader donut system, for example, although governance polls are done via coin vote, the mechanism that determines how many donuts (ie. coins) you get in the first place is based on reddit accounts: 1 upvote from 1 reddit account = \(n\) donuts earned. the ideal goal of an identity system is to make it relatively easy for individuals to get one identity, but relatively difficult to get many identities. in the /r/ethtrader donut system, that's reddit accounts, in the gitcoin clr matching gadget, it's github accounts that are used for the same purpose. but identity, at least the way it has been implemented so far, is a fragile thing.... oh, are you too lazy to make a big rack of phones? well maybe you're looking for this: usual warning about how sketchy sites may or may not scam you, do your own research, etc. etc. applies. arguably, attacking these mechanisms by simply controlling thousands of fake identities like a puppetmaster is even easier than having to go through the trouble of bribing people. and if you think the response is to just increase security to go up to government-level ids? well, if you want to get a few of those you can start exploring here, but keep in mind that there are specialized criminal organizations that are well ahead of you, and even if all the underground ones are taken down, hostile governments are definitely going to create fake passports by the millions if we're stupid enough to create systems that make that sort of activity profitable. and this doesn't even begin to mention attacks in the opposite direction, identity-issuing institutions attempting to disempower marginalized communities by denying them identity documents... collusion given that so many mechanisms seem to fail in such similar ways once multiple identities or even liquid markets get into the picture, one might ask, is there some deep common strand that causes all of these issues? i would argue the answer is yes, and the "common strand" is this: it is much harder, and more likely to be outright impossible, to make mechanisms that maintain desirable properties in a model where participants can collude, than in a model where they can't. most people likely already have some intuition about this; specific instances of this principle are behind well-established norms and often laws promoting competitive markets and restricting price-fixing cartels, vote buying and selling, and bribery. but the issue is much deeper and more general. in the version of game theory that focuses on individual choice that is, the version that assumes that each participant makes decisions independently and that does not allow for the possibility of groups of agents working as one for their mutual benefit, there are mathematical proofs that at least one stable nash equilibrium must exist in any game, and mechanism designers have a very wide latitude to "engineer" games to achieve specific outcomes. but in the version of game theory that allows for the possibility of coalitions working together, called cooperative game theory, there are large classes of games that do not have any stable outcome that a coalition cannot profitably deviate from. majority games, formally described as games of \(n\) agents where any subset of more than half of them can capture a fixed reward and split it among themselves, a setup eerily similar to many situations in corporate governance, politics and many other situations in human life, are part of that set of inherently unstable games. that is to say, if there is a situation with some fixed pool of resources and some currently established mechanism for distributing those resources, and it's unavoidably possible for 51% of the participants can conspire to seize control of the resources, no matter what the current configuration is there is always some conspiracy that can emerge that would be profitable for the participants. however, that conspiracy would then in turn be vulnerable to potential new conspiracies, possibly including a combination of previous conspirators and victims... and so on and so forth. round a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 this fact, the instability of majority games under cooperative game theory, is arguably highly underrated as a simplified general mathematical model of why there may well be no "end of history" in politics and no system that proves fully satisfactory; i personally believe it's much more useful than the more famous arrow's theorem, for example. there are two ways to get around this issue. the first is to try to restrict ourselves to the class of games that are "identity-free" and "collusion-safe", so where we do not need to worry about either bribes or identities. the second is to try to attack the identity and collusion resistance problems directly, and actually solve them well enough that we can implement non-collusion-safe games with the richer properties that they offer. identity-free and collusion-safe game design the class of games that is identity-free and collusion-safe is substantial. even proof of work is collusion-safe up to the bound of a single actor having ~23.21% of total hashpower, and this bound can be increased up to 50% with clever engineering. competitive markets are reasonably collusion-safe up until a relatively high bound, which is easily reached in some cases but in other cases is not. in the case of governance and content curation (both of which are really just special cases of the general problem of identifying public goods and public bads) a major class of mechanism that works well is futarchy typically portrayed as "governance by prediction market", though i would also argue that the use of security deposits is fundamentally in the same class of technique. the way futarchy mechanisms, in their most general form, work is that they make "voting" not just an expression of opinion, but also a prediction, with a reward for making predictions that are true and a penalty for making predictions that are false. for example, my proposal for "prediction markets for content curation daos" suggests a semi-centralized design where anyone can upvote or downvote submitted content, with content that is upvoted more being more visible, where there is also a "moderation panel" that makes final decisions. for each post, there is a small probability (proportional to the total volume of upvotes+downvotes on that post) that the moderation panel will be called on to make a final decision on the post. if the moderation panel approves a post, everyone who upvoted it is rewarded and everyone who downvoted it is penalized, and if the moderation panel disapproves a post the reverse happens; this mechanism encourages participants to make upvotes and downvotes that try to "predict" the moderation panel's judgements. another possible example of futarchy is a governance system for a project with a token, where anyone who votes for a decision is obligated to purchase some quantity of tokens at the price at the time the vote begins if the vote wins; this ensures that voting on a bad decision is costly, and in the limit if a bad decision wins a vote everyone who approved the decision must essentially buy out everyone else in the project. this ensures that an individual vote for a "wrong" decision can be very costly for the voter, precluding the possibility of cheap bribe attacks. a graphical description of one form of futarchy, creating two markets representing the two "possible future worlds" and picking the one with a more favorable price. source this post on ethresear.ch however, that range of things that mechanisms of this type can do is limited. in the case of the content curation example above, we're not really solving governance, we're just scaling the functionality of a governance gadget that is already assumed to be trusted. one could try to replace the moderation panel with a prediction market on the price of a token representing the right to purchase advertising space, but in practice prices are too noisy an indicator to make this viable for anything but a very small number of very large decisions. and often the value that we're trying to maximize is explicitly something other than maximum value of a coin. let's take a more explicit look at why, in the more general case where we can't easily determine the value of a governance decision via its impact on the price of a token, good mechanisms for identifying public goods and bads unfortunately cannot be identity-free or collusion-safe. if one tries to preserve the property of a game being identity-free, building a system where identities don't matter and only coins do, there is an impossible tradeoff between either failing to incentivize legitimate public goods or over-subsidizing plutocracy. the argument is as follows. suppose that there is some author that is producing a public good (eg. a series of blog posts) that provides value to each member of a community of 10000 people. suppose there exists some mechanism where members of the community can take an action that causes the author to receive a gain of $1. unless the community members are extremely altruistic, for the mechanism to work the cost of taking this action must be much lower than $1, as otherwise the portion of the benefit captured by the member of the community supporting the author would be much smaller than the cost of supporting the author, and so the system collapses into a tragedy of the commons where no one supports the author. hence, there must exist a way to cause the author to earn $1 at a cost much less than $1. but now suppose that there is also a fake community, which consists of 10000 fake sockpuppet accounts of the same wealthy attacker. this community takes all of the same actions as the real community, except instead of supporting the author, they support another fake account which is also a sockpuppet of the attacker. if it was possible for a member of the "real community" to give the author $1 at a personal cost of much less than $1, it's possible for the attacker to give themselves $1 at a cost much less than $1 over and over again, and thereby drain the system's funding. any mechanism that can help genuinely under-coordinated parties coordinate will, without the right safeguards, also help already coordinated parties (such as many accounts controlled by the same person) over-coordinate, extracting money from the system. a similar challenge arises when the goal is not funding, but rather determining what content should be most visible. what content do you think would get more dollar value supporting it: a legitimately high quality blog article benefiting thousands of people but benefiting each individual person relatively slightly, or this? or perhaps this? those who have been following recent politics "in the real world" might also point out a different kind of content that benefits highly centralized actors: social media manipulation by hostile governments. ultimately, both centralized systems and decentralized systems are facing the same fundamental problem, which is that the "marketplace of ideas" (and of public goods more generally) is very far from an "efficient market" in the sense that economists normally use the term, and this leads to both underproduction of public goods even in "peacetime" but also vulnerability to active attacks. it's just a hard problem. this is also why coin-based voting systems (like bihu's) have one major genuine advantage over identity-based systems (like the gitcoin clr or the /r/ethtrader donut experiment): at least there is no benefit to buying accounts en masse, because everything you do is proportional to how many coins you have, regardless of how many accounts the coins are split between. however, mechanisms that do not rely on any model of identity and only rely on coins fundamentally cannot solve the problem of concentrated interests outcompeting dispersed communities trying to support public goods; an identity-free mechanism that empowers distributed communities cannot avoid over-empowering centralized plutocrats pretending to be distributed communities. but it's not just identity issues that public goods games are vulnerable too; it's also bribes. to see why, consider again the example above, but where instead of the "fake community" being 10001 sockpuppets of the attacker, the attacker only has one identity, the account receiving funding, and the other 10000 accounts are real users but users that receive a bribe of $0.01 each to take the action that would cause the attacker to gain an additional $1. as mentioned above, these bribes can be highly obfuscated, even through third-party custodial services that vote on a user's behalf in exchange for convenience, and in the case of "coin vote" designs an obfuscated bribe is even easier: one can do it by renting coins on the market and using them to participate in votes. hence, while some kinds of games, particularly prediction market or security deposit based games, can be made collusion-safe and identity-free, generalized public goods funding seems to be a class of problem where collusion-safe and identity-free approaches unfortunately just cannot be made to work. collusion resistance and identity the other alternative is attacking the identity problem head-on. as mentioned above, simply going up to higher-security centralized identity systems, like passports and other government ids, will not work at scale; in a sufficiently incentivized context, they are very insecure and vulnerable to the issuing governments themselves! rather, the kind of "identity" we are talking about here is some kind of robust multifactorial set of claims that an actor identified by some set of messages actually is a unique individual. a very early proto-model of this kind of networked identity is arguably social recovery in htc's blockchain phone: the basic idea is that your private key is secret-shared between up to five trusted contacts, in such a way that mathematically ensures that three of them can recover the original key, but two or fewer can't. this qualifies as an "identity system" it's your five friends determining whether or not someone trying to recover your account actually is you. however, it's a special-purpose identity system trying to solve a problem personal account security that is different from (and easier than!) the problem of attempting to identify unique humans. that said, the general model of individuals making claims about each other can quite possibly be bootstrapped into some kind of more robust identity model. these systems could be augmented if desired using the "futarchy" mechanic described above: if someone makes a claim that someone is a unique human, and someone else disagrees, and both sides are willing to put down a bond to litigate the issue, the system can call together a judgement panel to determine who is right. but we also want another crucially important property: we want an identity that you cannot credibly rent or sell. obviously, we can't prevent people from making a deal "you send me $50, i'll send you my key", but what we can try to do is prevent such deals from being credible make it so that the seller can easily cheat the buyer and give the buyer a key that doesn't actually work. one way to do this is to make a mechanism by which the owner of a key can send a transaction that revokes the key and replaces it with another key of the owner's choice, all in a way that cannot be proven. perhaps the simplest way to get around this is to either use a trusted party that runs the computation and only publishes results (along with zero knowledge proofs proving the results, so the trusted party is trusted only for privacy, not integrity), or decentralize the same functionality through multi-party computation. such approaches will not solve collusion completely; a group of friends could still come together and sit on the same couch and coordinate votes, but they will at least reduce it to a manageable extent that will not lead to these systems outright failing. there is a further problem: initial distribution of the key. what happens if a user creates their identity inside a third-party custodial service that then stores the private key and uses it to clandestinely make votes on things? this would be an implicit bribe, the user's voting power in exchange for providing to the user a convenient service, and what's more, if the system is secure in that it successfully prevents bribes by making votes unprovable, clandestine voting by third-party hosts would also be undetectable. the only approach that gets around this problem seems to be.... in-person verification. for example, one could have an ecosystem of "issuers" where each issuer issues smart cards with private keys, which the user can immediately download onto their smartphone and send a message to replace the key with a different key that they do not reveal to anyone. these issuers could be meetups and conferences, or potentially individuals that have already been deemed by some voting mechanic to be trustworthy. building out the infrastructure for making collusion-resistant mechanisms possible, including robust decentralized identity systems, is a difficult challenge, but if we want to unlock the potential of such mechanisms, it seems unavoidable that we have to do our best to try. it is true that the current computer-security dogma around, for example, introducing online voting is simply "don't", but if we want to expand the role of voting-like mechanisms, including more advanced forms such as quadratic voting and quadratic finance, to more roles, we have no choice but to confront the challenge head-on, try really hard, and hopefully succeed at making something secure enough, for at least some use cases. dark mode toggle the triangle of harm 2017 jul 16 see all posts the following is a diagram from a slide that i made in one of my presentations at cornell this week: if there was one diagram that could capture the core principle of casper's incentivization philosophy, this might be it. hence, it warrants some further explanation. the diagram shows three constituencies the minority, the majority and the protocol (ie. users), and four arrows representing possible adversarial actions: the minority attacking the protocol, the minority attacking the majority, the majority attacking the protocol, and the majority attacking the minority. examples of each include: minority attacking the protocol finney attacks (an attack done by a miner on a proof of work blockchain where the miner double-spends unconfirmed, or possibly single-confirmed, transactions) minority attacking the majority feather forking (a minority in a proof of work chain attempting to revert any block that contains some undesired transactions, though giving up if the block gets two confirmations) majority attacking the protocol traditional 51% attacks majority attacking the minority a 51% censorship attack, where a cartel refuses to accept any blocks from miners (or validators) outside the cartel the essence of casper's philosophy is this: for all four categories of attack, we want to put an upper bound on the ratio between the amount of harm suffered by the victims of the attack and the cost to the attacker. in some ways, every design decision in casper flows out of this principle. this differs greatly from the usual proof of work incentivization school of thought in that in the proof of work view, the last two attacks are left undefended against. the first two attacks, finney attacks and feather forking, are costly because the attacker risks their blocks not getting included into the chain and so loses revenue. if the attacker is a majority, however, the attack is costless, because the attacker can always guarantee that their chain will be the main chain. in the long run, difficulty adjustment ensures that the total revenue of all miners is exactly the same no matter what, and this further means that if an attack causes some victims to lose money, then the attacker gains money. this property of proof of work arises because traditional nakamoto proof of work fundamentally punishes dissent if you as a miner make a block that aligns with the consensus, you get rewarded, and if you make a block that does not align with the consensus you get penalized (the penalty is not in the protocol; rather, it comes from the fact that such a miner expends electricity and capital to produce the block and gets no reward). casper, on the other hand, works primarily by punishing equivocation if you send two messages that conflict with each other, then you get very heavily penalized, even if one of those messages aligns with the consensus (read more on this in the blog post on "minimal slashing conditions"). hence, in the event of a finality reversion attack, those who caused the reversion event are penalized, and everyone else is left untouched; the majority can attack the protocol only at heavy cost, and the majority cannot cause the minority to lose money. it gets more challenging when we move to talking about two other kinds of attacks liveness faults, and censorship. a liveness fault is one where a large portion of casper validators go offline, preventing the consensus from reaching finality, and a censorship fault is one where a majority of casper validators refuse to accept some transactions, or refuse to accept consensus messages from other casper validators (the victims) in order to deprive them of rewards. this touches on a fundamental dichotomy: speaker/listener fault equivalence. suppose that person b says that they did not receive a message from person a. there are two possible explanations: (i) person a did not send the message, (ii) person b pretended not to hear the message. given just the evidence of b's claim, there is no way to tell which of the two explanations is correct. the relation to blockchain protocol incentivization is this: if you see a protocol execution where 70% of validators' messages are included in the chain and 30% are not, and see nothing else (and this is what the blockchain sees), then there is no way to tell whether the problem is that 30% are offline or 70% are censoring. if we want to make both kinds of attacks expensive, there is only one thing that we can do: penalize both sides. penalizing both sides allows either side to "grief" the other, by going offline if they are a minority and censoring if they are a majority. however, we can establish bounds on how easy this griefing is, through the technique of griefing factor analysis. the griefing factor of a strategy is essentially the amount of money lost by the victims divided by the amount of money lost by the attackers, and the griefing factor of a protocol is the highest griefing factor that it allows. for example, if a protocol allows me to cause you to lose $3 at a cost of $1 to myself, then the griefing factor is 3. if there are no ways to cause others to lose money, the griefing factor is zero, and if you can cause others to lose money at no cost to yourself (or at a benefit to yourself), the griefing factor is infinity. in general, wherever a speaker/listener dichotomy exists, the griefing factor cannot be globally bounded above by any value below 1. the reason is simple: either side can grief the other, so if side \(a\) can grief side \(b\) with a factor of \(x\) then side \(b\) can grief side \(a\) with a factor of \(\frac{1}{x}\); \(x\) and \(\frac{1}{x}\) cannot both be below 1 simultaneously. we can play around with the factors; for example, it may be considered okay to allow griefing factors of 2 for majority attackers in exchange for keeping the griefing factor at 0.5 for minorities, with the reasoning that minority attackers are more likely. we can also allow griefing factors of 1 for small-scale attacks, but specifically for large-scale attacks force a chain split where on one chain one side is penalized and on another chain another side is penalized, trusting the market to pick the chain where attackers are not favored. hence there is a lot of room for compromise and making tradeoffs between different concerns within this framework. penalizing both sides has another benefit: it ensures that if the protocol is harmed, the attacker is penalized. this ensures that whoever the attacker is, they have an incentive to avoid attacking that is commensurate with the amount of harm done to the protocol. however, if we want to bound the ratio of harm to the protocol over cost to attackers, we need a formalized way of measuring how much harm to the protocol was done. this introduces the concept of the protocol utility function a formula that tells us how well the protocol is doing, that should ideally be calculable from inside the blockchain. in the case of a proof of work chain, this could be the percentage of all blocks produced that are in the main chain. in casper, protocol utility is zero for a perfect execution where every epoch is finalized and no safety failures ever take place, with some penalty for every epoch that is not finalized, and a very large penalty for every safety failure. if a protocol utility function can be formalized, then penalties for faults can be set as close to the loss of protocol utility resulting from those faults as possible. dark mode toggle zmowa 2000 jan 01 see all posts specjalne podziękowania dla glena weyla, phila daiana i jinglana wanga za recenzję, i dla dawida kurdziela i bichvan2408 za tłumaczenie w ciągu ostatnich kilku lat wzrosło zainteresowanie wykorzystaniem rozmyślnie skonstruowanych bodźców ekonomicznych i mechanizmów zaprojektowanych w celu dostosowania zachowania uczestników w różnych kontekstach. w przestrzeni blockchain konstrukcja mechanizmu przede wszystkim zapewnia bezpieczeństwo samego łańcucha bloków, zachęcając górników lub walidatorów proof of stake do uczciwego udziału, ale ostatnio jest on stosowany w rynkach prognoz, „tokenowo nadzorowanych rejestrach" i wielu innych kontekstach. powstający ruch radicalxchange zainicjował w międzyczasie eksperymenty z podatkami harbergera, głosowaniem kwadratowym, finansowaniem kwadratowym i in. w ostatnim czasie wzrosło również zainteresowanie wykorzystaniem bodźców opartych na tokenach w celu promowania wysokiej jakości postów w mediach społecznościowych. jednakże w miarę jak rozwój tych systemów przechodzi z teorii do praktyki, pojawia się szereg wyzwań, którym należy stawić czoła, a których, jak sądzę, jeszcze wystarczająco nie rozpoznano. świeżym przykładem tego przejścia od teorii do wdrożenia jest bihu, chińska platforma, która niedawno wypuściła oparty na monetach mechanizm zachęcania ludzi do pisania postów. podstawowym mechanizmem (patrz white paper w języku chińskim tutaj) jest to, że jeśli użytkownik platformy posiada żetony key, ma możliwość postawienia tych żetonów na artykuły; każdy użytkownik może oddać k „głosów poparcia" dziennie, a „waga" każdego głosu poparcia jest proporcjonalna do stawki użytkownika, który oddał głos. artykuły z większą ilością udziałów w głosach poparcia są bardziej widoczne, a autor artykułu otrzymuje nagrodę w postaci tokenów key mniej więcej proporcjonalną do ilości głosów key popierających ten artykuł. jest to wielkie uproszczenie, a rzeczywisty mechanizm ma w sobie pewne nieliniowości, ale nie są one niezbędne dla podstawowego funkcjonowania mechanizmu. key ma wartość, ponieważ może być używany na różne sposoby wewnątrz platformy, ale przede wszystkim procent wszystkich przychodów z reklam wykorzystywany jest do zakupu i spalania key (hura, wielkie brawa dla nich za to, a nie robienie jeszcze jednego medium do wymiany tokenów!). ten rodzaj konstrukcji jest daleki od unikalności; zachęcanie do tworzenia treści online jest czymś, na czym bardzo wielu ludziom zależy i było wiele projektów o podobnym charakterze, jak również kilka dość odmiennych. i w tym przypadku ta konkretna platforma jest już w znacznym stopniu wykorzystywana: kilka miesięcy temu, handlujący ethereum subreddit /r/ethtrader wprowadził nieco podobną eksperymentalną funkcję, dzięki której token o nazwie „pączek" jest wydawany użytkownikom wyrażającym komentarze otrzymujące głosy poparcia, z ustaloną ilością pączków wydawanych tygodniowo użytkownikom proporcjonalnie do tego, ile głosów poparcia otrzymał ich komentarz. pączki mogą być wykorzystane do zakupu prawa do umieszczenia treści bannera na górze subredditu, a także do głosowania w sondażach społecznościowych. jednakże, w przeciwieństwie do tego, co dzieje się w systemie key, tutaj nagroda, którą otrzymuje b, gdy a zagłosuje na b, nie jest proporcjonalna do istniejącej liczby monet należących do a; zamiast tego, każde konto reddit ma równą możliwość wpłaty na inne konta reddit. tego rodzaju eksperymenty próbujące nagradzać tworzenie treści wysokiej jakości w sposób wykraczający poza znane ograniczenia darowizn/mikropłatności są bardzo cenne. niedocenianie treści internetowych generowanych przez użytkowników jest istotnym problemem w społeczeństwie w ogóle (patrz „radykalizm liberalny" i „dane jako praca"). otuchy dodaje fakt, że kryptospołeczności próbują wykorzystać siłę konstrukcji mechanizmu, by wejść na drogę do jego rozwiązania. ale niestety, te systemy są również podatne na atak. samogłosowanie, plutokracja i łapówki oto jak można ekonomicznie zaatakować zaproponowany powyżej projekt. załóżmy, że jakiś zamożny użytkownik nabywa pewną ilość n tokenów, a w rezultacie każdy z jego k głosów daje odbiorcy nagrodę w wysokości n * q (q tutaj prawdopodobnie jest to bardzo mała liczba, np. q = 0,000001). użytkownik po prostu głosuje na swoje własne konta kukiełkowe, dając sobie nagrodę w wysokości n * k * q. następnie system po prostu się załamuje, a każdy użytkownik otrzymuje „stopę procentową" w wysokości k * q za okres, a mechanizm nie osiąga nic innego. rzeczywisty mechanizm bihu zdawał się to przewidywać i ma pewną superliniową logikę, w której artykuły z większą ilością głosów poparcia key uzyskują nieproporcjonalnie większą nagrodę, pozornie zachęcając raczej do poparcia popularnych postów niż do samodzielnego głosowania. powszechnym wzorcem wśród systemów zarządzania głosowaniem za pomocą monet jest dodawanie tego rodzaju superliniowości, aby zapobiec osłabianiu całego systemu przez samogłosowanie. większość systemów dpos posiada ograniczoną liczbę slotów dla delegatów z zerowymi nagrodami dla każdego, kto nie otrzyma wystarczającej liczby głosów, aby dołączyć do jednego z slotów, z podobnym skutkiem. jednak systemy te niezmiennie wprowadzają dwie nowe słabości: subsydiują plutokrację, ponieważ bardzo zamożne osoby i kartele nadal mogą uzyskać wystarczająco dużo środków, aby samodzielnie głosować. mogą one zostać ominięte przez użytkowników *** przekupujących*** innych użytkowników, aby głosować na nich masowo. ataki łapówkarskie mogą brzmieć zbyt przesadnie (kto tu kiedykolwiek w życiu przyjął łapówkę?), ale w dojrzałym ekosystemie są o wiele bardziej realistyczne, niż się wydaje. w większości kontekstów, w których łapówki miały miejsce w przestrzeni blockchain, operatorzy używają eufemistycznej nowej nazwy, aby nadać koncepcji przyjazną twarz: to nie jest łapówka, to „staking pool", która „rozdziela dywidendę". łapówki można nawet zatuszować: wyobraź sobie kantor wymiany kryptowalut, który oferuje zerowe opłaty i wydaje pieniądze na niezwykle dobry interfejs użytkownika i nawet nie próbuje zebrać zysku. zamiast tego używa monet, które użytkownicy wpłacają, aby uczestniczyć w różnych systemach głosowania za pomocą monet. nieuchronnie pojawią się również ludzie, którzy będą postrzegać zmowę w grupie jako zwykłą, normalną rzecz. przykładem może być niedawny skandal z udziałem eos dpos: w końcu istnieje możliwość „negatywnej łapówki", tzn. szantażu lub przymusu, grożąc uczestnikom krzywdą, chyba że zadziałają oni w określony sposób wewnątrz mechanizmu. w eksperymencie /r/ethtrader strach przed ludźmi przychodzącymi i kupującymi pączki w celu zmiany sondaży zarządzających doprowadził do tego, że społeczność zdecydowała się na stworzenie tylko zablokowanych (tj. niepodlegających wymianie handlowej) pączków uprawnionych do wykorzystania w głosowaniu. ale jest jeszcze tańszy atak niż kupowanie pączków (atak, który można uznać za rodzaj zamaskowanej łapówki): wynajmowanie ich. jeśli napastnik posiada już eth, może użyć go jako zabezpieczenia na platformie takiej jak compound, aby pożyczyć jakiś token, dając ci pełne prawo do użycia tego tokena w dowolnym celu, włączając w to udział w głosowaniu. kiedy to zrobi, po prostu odsyła tokeny z powrotem do kontraktu pożyczki, aby odzyskać swoje zabezpieczenie wszystko to bez konieczności znoszenia nawet jednej sekundy ekspozycji cenowej na token, którego użył do oddania głosu monetą, nawet jeśli mechanizm głosowania monetą zawiera blokadę czasową (jak np. bihu). w każdym przypadku problemy związane z łapówkarstwem i przypadkowe nadmierne wzmocnienie dobrze powiązanych i zamożnych uczestników okazują się zaskakująco trudne do uniknięcia. tożsamość niektóre systemy próbują złagodzić plutokratyczne aspekty głosowania monetami poprzez wykorzystanie systemu tożsamości. w przypadku systemu pączków /r/ethtrader, na przykład, sondaże zarządzające są wprawdzie przeprowadzane za pomocą głosowania monetami, jednak mechanizm, który decyduje o tym, jak dużo pączków (tj. monet) otrzymujesz w pierwszej kolejności oparty jest na kontach reddit: 1 głos poparcia z 1 konta reddit = n zarobionych pączków. idealnym celem systemu tożsamości jest sprawienie, by uzyskanie przez jednostkę jednej tożsamości było stosunkowo łatwe, natomiast uzyskanie wielu tożsamości stosunkowo trudne. w systemie pączków /r/ethtrader są to konta reddit, a w gadżecie dopasowującym gitcoin clr używane są konta githuba. ale tożsamość, przynajmniej dotychczas implementowana, jest delikatną rzeczą... och, jesteś zbyt leniwy, żeby zrobić duży stojak z telefonami? cóż, może szukasz tego: zwykłe ostrzeżenie o możliwych nieuczciwych praktykach na tych stronach jest tego warte: przeprowadzaj własne badania i zachowaj czujność. prawdopodobnie, atakowanie tych mechanizmów poprzez zwykłe kontrolowanie tysięcy fałszywych tożsamości jak władca marionetek jest nawet łatwiejsze niż konieczność przekupywania ludzi. a jeśli uważasz, że odpowiedzią jest po prostu zwiększenie bezpieczeństwa do poziomu rządowych dokumentów tożsamości? cóż, jeśli chcesz zdobyć kilka z nich, możesz zacząć poszukiwania tutaj ale pamiętaj, że istnieją wyspecjalizowane organizacje przestępcze, które są daleko przed tobą. nawet jeśli wszystkie struktury przestępcze zostałyby rozbite, istnieją wrogie rządy, które na pewno stworzą miliony sfałszowanych paszportów, jeśli będziemy na tyle głupi, by stworzyć systemy, które sprawią, że tego rodzaju działalność będzie opłacalna. i nie obejmuje to nawet ataków w odwrotnym kierunku, gdy instytucje wydające dokumenty tożsamości próbują zepchnąć na margines społeczności poprzez odmawianie im dokumentów tożsamości... zmowa biorąc pod uwagę, że tak wiele mechanizmów wydaje się podobnie zawodzić, kiedy w grę wchodzi wiele tożsamości lub nawet płynnych rynków, można zapytać, czy istnieje jakiś głęboki wspólny aspekt, który powoduje te wszystkie problemy? twierdzę, że odpowiedź brzmi „tak", a „wspólny aspekt" jest następujący: o wiele trudniejsze, a zapewne wręcz niemożliwe jest stworzenie mechanizmów, które zachowują pożądane właściwości w modelu, w którym uczestnicy mogą się zmawiać, niż w modelu, w którym nie mogą. większość ludzi prawdopodobnie ma już pewne przeczucia na ten temat; konkretne realizacje tej zasady są źródłem dobrze ugruntowanych norm i często przepisów prawa promujących konkurencję na rynku i ograniczających działania karteli ustalających ceny, kupowanie i sprzedawanie głosów oraz przekupstwo. ale problem jest o wiele głębszy i bardziej powszechny. w tej wersji teorii gier, która skupia się na indywidualnym wyborze czyli w wersji, która zakłada, że każdy uczestnik podejmuje decyzje samodzielnie i która nie dopuszcza możliwości pracy grup agentów działających wspólnie dla wzajemnych korzyści, istnieją matematyczne dowody, że w każdej grze musi istnieć przynajmniej jedna stabilna równowaga nasha, a projektanci mechanizmów mają bardzo dużą swobodę w „konstruowaniu" gier, aby osiągnąć konkretne wyniki. ale w wersji teorii gier, która dopuszcza możliwość koalicji pracujących razem, zwanej teorią gier kooperacyjnych, istnieją duże klasy gier które nie mają stabilnego wyniku, od którego koalicja nie może się zyskownie różnić. gry większościowe formalnie opisane jako gry n agentów, gdzie dowolny podzbiór ponad połowy z nich może zdobyć stałą nagrodę i podzielić ją między sobą; konfiguracja bardzo podobna do wielu sytuacji w zakresie ładu korporacyjnego, polityki i wielu innych sytuacji z życia człowieka są częścią tego zbioru z natury niestabilnych gier. oznacza to, że jeśli istnieje sytuacja z pewną stałą pulą zasobów i pewnym obecnie ustalonym mechanizmem ich dystrybucji, a 51% uczestników może nieuchronnie spiskować w celu przejęcia kontroli nad zasobami, bez względu na to, jaka jest obecna konfiguracja, zawsze może pojawić się jakiś spisek, który byłby korzystny dla uczestników. spisek ten byłby jednak z kolei narażony na potencjalne nowe konspiracje, w tym być może kombinację poprzednich spiskowców i ofiar... i tak dalej i tak dalej. runda a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 ten fakt, niestabilność gier większościowych w ramach teorii gier kooperacyjnych, jest prawdopodobnie bardzo niedoceniany jako uproszczony ogólny model matematyczny wyjaśniający, dlaczego w polityce nie może być żadnego „końca historii" i żadnego systemu, który okazałby się w pełni zadowalający. osobiście uważam, że jest on o wiele bardziej użyteczny niż bardziej znane twierdzenie arrowa, na przykład. są dwa sposoby na obejście tego problemu. pierwszym z nich jest próba ograniczenia nas samych do klasy gier, które są „wolne od tożsamości" i „odporne na zmowy", żebyśmy nie musieli martwić się ani o łapówki, ani o tożsamość. drugim jest bezpośrednie zajęcie się problemami tożsamości i odporności na zmowy i rzeczywiste rozwiązanie ich wystarczająco dobrze, aby wdrożyć gry bez zmowy, oferujące lepsze funkcje. projekt gry bez tożsamości i odpornej na zmowy.klasa gier wolnych od tożsamości i odpornych na zmowy ma zasadnicze znaczenie. nawet proof of work jest odporny na zmowy aż do momentu, w którym jeden aktor ma ~23,21% całkowitej mocy haszowania, a ta wartość może być zwiększona do 50% przy użyciu sprytnej inżynierii. rynki konkurencyjne są całkiem odporne na zmowy do stosunkowo wysokiej wartości granicznej, która w niektórych przypadkach jest łatwo osiągalna, a w innych nie. w przypadku nadzorowania i kontroli treści (które tak naprawdę są tylko szczególnymi przypadkami ogólnego problemu identyfikacji publicznych dóbr i publicznych krzywd) główną klasą dobrze funkcjonującego mechanizmu jest futarchia zazwyczaj przedstawiana jako „zarządzanie przez rynek prognostyczny", choć twierdzę również, że korzystanie z depozytów zabezpieczających jest zasadniczo w tej samej klasie technologii. sposób, w jaki działają mechanizmy futarchii, w swojej najbardziej ogólnej formie, polega na tym, że „głosowanie" jest nie tylko wyrażeniem opinii, ale także przewidywaniem, z nagrodą za tworzenie prawdziwych przepowiedni i karą za tworzenie fałszywych. na przykład moja propozycja dla „rynków przewidywań dla dao nadzorujących treści" sugeruje na wpół scentralizowaną konstrukcję, gdzie każdy może zagłosować lub odrzucić nadesłane treści, z treścią na którą oddaje się więcej głosów poparcia, gdzie istnieje również „panel moderacyjny", który podejmuje ostateczne decyzje. dla każdego wpisu istnieje małe prawdopodobieństwo (proporcjonalne do całkowitej liczby głosów za i przeciw na dany wpis), że panel moderacyjny zostanie wezwany do podjęcia ostatecznej decyzji w sprawie wpisu. jeśli panel moderacyjny zatwierdzi dany wpis, każdy, kto go na niego głosował jest nagradzany, a każdy kto głosował przeciw, jest karany, a jeśli panel moderacyjny nie zatwierdzi wpisu, następuje sytuacja odwrotna; mechanizm ten zachęca uczestników do wymyślenia głosów, które starają się „przewidzieć" osąd panelu moderacyjnego. innym możliwym przykładem futarchii jest system zarządzania projektem z użyciem tokena, w którym każdy, kto głosuje za decyzją, jest zobowiązany do zakupu pewnej ilości tokenów po cenie w momencie rozpoczęcia głosowania, jeśli głosowanie wygra. gwarantuje to, że głosowanie za złą decyzją jest kosztowne, a w skrajnych przypadkach, jeśli zła decyzja wygra głosowanie, każdy kto ją zatwierdził, musi zasadniczo wykupić wszystkich innych w projekcie. gwarantuje to, że indywidualne głosowanie za „niewłaściwą" decyzją może być bardzo kosztowne dla wyborcy, wykluczając możliwość tanich ataków łapówkarskich. opis graficzny jednej z form futarchii, tworzącej dwa rynki reprezentujące dwa „możliwe przyszłe światy" i wybierającej ten, który ma korzystniejszą cenę. źródło ten post na ethresear.ch jednak zakres tego, co tego typu mechanizmy mogą robić, jest ograniczony. w powyższym przykładzie nadzoru treści nie rozwiązujemy tak naprawdę kwestii zarządzania, po prostu skalujemy funkcjonalność gadżetu zarządzającego, który już został uznany za godny zaufania. można by spróbować zastąpić panel moderacyjny rynkiem predykcyjnym w oparciu o cenę tokena reprezentującego prawo do zakupu powierzchni reklamowej, ale w praktyce ceny są zbyt hałaśliwym wskaźnikiem, aby było to wykonalne dla czegokolwiek oprócz bardzo małej liczby bardzo dużych decyzji. i często wartość, którą staramy się zmaksymalizować, jest wyraźnie czymś innym niż maksymalna wartość monety. przyjrzyjmy się dokładniej, dlaczego w bardziej ogólnym przypadku, gdy nie możemy łatwo określić wartości decyzji dotyczącej zarządzania poprzez jej wpływ na cenę tokena, dobre mechanizmy identyfikacji dóbr i krzywd publicznych nie mogą być niestety wolne od tożsamości lub odporne na zmowy. jeśli ktoś próbuje zachować własność gry wolnej od tożsamości, budując system, w którym tożsamość nie ma znaczenia i mają ją tylko monety, istnieje niemożliwy do osiągnięcia kompromis pomiędzy brakiem zachęt do korzystania z legalnych dóbr publicznych, a nadmiernym subsydiowaniem plutokracji. argumentacja jest następująca. przypuśćmy, że jest jakiś autor, który wytwarza dobro publiczne (np. serię postów na blogu), które zapewnia wartość każdemu członkowi społeczności liczącej 10 000 osób. przypuśćmy, że istnieje pewien mechanizm, w którym członkowie społeczności mogą podjąć działanie powodujące, że autor otrzymuje zysk w wysokości 1\(. o ile członkowie społeczności nie są *niezwykle* altruistyczni, to aby mechanizm działał, koszt podjęcia tej akcji musi być znacznie niższy niż 1\), ponieważ w przeciwnym razie część korzyści odniesionych przez członka społeczności wspierającego autora byłaby znacznie mniejsza niż koszt wsparcia autora, a więc system popada w tragedię wspólnoty, gdzie nikt nie wspiera autora. dlatego musi istnieć sposób na to, aby autor zarobił 1$ przy koszcie znacznie mniejszym niż 1\(. ale teraz załóżmy, że istnieje również fałszywa społeczność, która składa się z 10 000 fałszywych kont marionetkowych tego samego bogatego atakującego. ta społeczność podejmuje wszystkie te same działania, co prawdziwa społeczność, z wyjątkiem tego, że zamiast wspierać autora, wspomaga *inne* fałszywe konto, które jest również marionetką atakującego. jeśli członek „prawdziwej społeczności" mógł dać autorowi 1\) przy osobistym koszcie znacznie mniejszym niż 1$, to jest możliwe by napastnik dał samemu sobie\(1 przy koszcie znacznie mniejszym niż 1\) za każdym razem, a tym samym wyczerpał fundusze systemu. każdy mechanizm, który może naprawdę pomóc słabo skoordynowanym stronom w koordynowaniu bez odpowiednich zabezpieczeń, pomoże również już skoordynowanym stronom (takim jak wiele kont kontrolowanych przez tę samą osobę) nadmiernie skoordynowanie, wydobywając pieniądze z systemu. podobne wyzwanie pojawia się, gdy celem nie jest finansowanie, ale raczej określenie, jaka treść powinna być najbardziej widoczna. jaka treść, twoim zdaniem, otrzymałaby większą wspierającą wartość dolarową: prawdziwie wysokiej jakości artykuł blogowy przynoszący korzyści tysiącom ludzi, choć każda osoba korzysta z niego stosunkowo nieznacznie, czy to? a może to? ci, którzy śledzili niedawne wydarzenia polityczne na świecie, mogą również wskazać na inny rodzaj treści, który przynosi korzyści wysoce scentralizowanym aktorom: manipulację mediami społecznościowymi przez wrogie rządy. ostatecznie zarówno systemy scentralizowane, jak i zdecentralizowane stoją przed tym samym fundamentalnym problemem, który polega na tym, że „rynek idei" (i ogólnie dóbr publicznych) jest bardzo daleki od „efektywnego rynku" w sensie zwykle używanym przez ekonomistów, a to prowadzi zarówno do niedoboru dóbr publicznych nawet w „czasie pokoju", jak i podatności na aktywne ataki. to po prostu poważny problem. dlatego też systemy głosowania oparte na monetach (takie jak bihu) mają jedną główną przewagę nad systemami opartymi na tożsamości (jak gitcoin clr lub eksperyment z pączkami /r/ethtrader): przynajmniej nie ma żadnej korzyści z masowego kupowania kont, ponieważ wszystko, co robisz, jest proporcjonalne do tego ile masz monet, niezależnie od tego, na ile kont są one podzielone. jednakże mechanizmy, które nie opierają się na żadnym modelu tożsamości i zasadniczo opierają się jedynie na monetach, nie mogą rozwiązać problemu skoncentrowanych interesów, które wygrywają z rozproszonymi społecznościami starającymi się wspierać dobra publiczne. wolny od tożsamości mechanizm, który wzmacnia rozproszone społeczności, nie może uniknąć nadmiernego wzmocnienia scentralizowanych plutokratów udających rozproszone społeczności. ale nie chodzi tylko o kwestie tożsamości, na które narażone są również gry związane z dobrami publicznymi; chodzi też o łapówki. aby zobaczyć dlaczego, rozważmy ponownie powyższy przykład, gdzie zamiast „fałszywej społeczności", którą jest 10001 marionetek atakującego, atakujący ma tylko jedną tożsamość, konto otrzymujące fundusze, a pozostałe 10 000 kont to prawdziwi użytkownicy ale użytkownicy otrzymujący łapówkę po 0,01$ każdy, za podjęcie akcji, która spowodowałaby, że atakujący zyskałby dodatkowy 1$. jak wspomniano powyżej, łapówki te mogą być w znacznym stopniu zatuszowane, nawet poprzez usługi powiernicze świadczone przez osoby trzecie, które głosują w imieniu użytkownika w zamian za wygody, a w przypadku modeli „głosujących monetami" zatuszowanie łapówki jest jeszcze łatwiejsze: można to zrobić poprzez wypożyczenie monet na rynku i wykorzystanie ich do udziału w głosowaniu. zatem, o ile niektóre rodzaje gier, w szczególności gier opartych na rynku przewidywań lub na depozytach zabezpieczających, można uczynić odpornymi na zmowy i pozbawionymi tożsamości, o tyle uogólnione finansowanie dóbr publicznych wydaje się klasą problemów, w której niestety nie da się zastosować podejścia odpornego na zmowy i pozbawionego tożsamości. odporność na zmowy i tożsamość inną alternatywą jest atakowanie problemu tożsamości bezpośrednio. jak wspomniano powyżej, samo przejście na scentralizowane systemy tożsamości o wyższym poziomie bezpieczeństwa, takie jak paszporty i inne rządowe dokumenty tożsamości, nie będzie działać na dużą skalę; w sytuacji wielu zachęt są one bardzo niepewne i podatne na zagrożenia ze strony samych rządów wydających dokumenty! rodzaj „tożsamości", o której tutaj mówimy, jest raczej pewnego rodzaju solidnym wieloczynnikowym zbiorem twierdzeń, że aktor identyfikowany przez pewien zbiór komunikatów jest w rzeczywistości niepowtarzalną jednostką. bardzo wczesnym prototypowym modelem tego rodzaju tożsamości sieciowej jest prawdopodobnie społeczny mechanizm odzyskiwania kluczy w blockchainowym telefonie htc: podstawową ideą jest to, że twój klucz prywatny jest podzielony w tajemnicy pomiędzy maksymalnie pięć zaufanych kontaktów, w taki sposób, że matematycznie zapewnia, iż trzy z nich mogą odzyskać oryginalny klucz, ale dwa lub mniej nie mogą. kwalifikuje się to jako „system tożsamości" to piątka twoich przyjaciół decyduje o tym, czy ktoś, kto próbuje odzyskać twoje konto, to ty, czy nie. jest to jednak system tożsamości specjalnego przeznaczenia, próbujący rozwiązać problem bezpieczeństwa konta osobistego który różni się od (i jest łatwiejszy niż!) problemu próby identyfikacji konkretnych osób. podsumowując, ogólny model konkretnych osób składających oświadczenia o sobie nawzajem może być prawdopodobnie powiązany z jakimś bardziej solidnym modelem tożsamości. systemy te mogą być w razie potrzeby rozszerzone przy użyciu opisanego powyżej mechanizmu „futarchii": jeśli ktoś twierdzi, że ktoś jest konkretną osobą, a ktoś inny się na to nie zgadza, a obie strony są skłonne nawiązać współpracę w celu rozstrzygnięcia sprawy, system może zwołać zespół orzekający w celu ustalenia, kto ma rację. ale chcemy też innej niezwykle istotnej własności: chcemy tożsamości, której nie można wiarygodnie wynająć ani sprzedać. oczywiście, nie możemy przeszkodzić ludziom w zawarciu umowy typu „wyślij mi 50 dolarów, a ja wyślę ci mój klucz", ale możemy spróbować zapobiegać wiarygodności takich transakcji zrobić tak, aby sprzedający mógł łatwo oszukać kupującego i dać mu klucz, który tak naprawdę nie działa. jednym ze sposobów dokonania tego jest stworzenie mechanizmu, dzięki któremu właściciel klucza może wysłać transakcję, która unieważnia klucz i zastępuje go innym, wybranym przez właściciela, a wszystko to w sposób, którego nie można udowodnić. być może najprostszym sposobem na obejście tego problemu jest albo użycie zaufanej strony, która przeprowadza obliczenia i publikuje tylko wyniki (wraz z dowodami z wiedzą zerową potwierdzającymi wyniki, więc zaufana strona jest zaufana tylko dla prywatności, a nie integralności), albo zdecentralizowanie tej samej funkcjonalności dzięki wielopodmiotowym obliczeniom. takie podejście nie rozwiąże całkowicie problemu zmowy; grupa przyjaciół mogłaby nadal spotykać się na tej samej kanapie i koordynować głosowania, ale przynajmniej ograniczy ją w możliwym do opanowania stopniu, który nie doprowadzi do całkowitego upadku tych systemów. jest jeszcze jeden problem: wstępna dystrybucja klucza. co się stanie, jeśli użytkownik stworzy swoją tożsamość w zewnętrznym serwisie powierniczym, który następnie przechowuje klucz prywatny i używa go do potajemnego głosowania? byłaby to ukryta łapówka, siła głosu użytkownika w zamian za dostarczenie użytkownikowi wygodnej usługi, a co więcej, jeśli system jest bezpieczny w tym sensie, że skutecznie zapobiega łapówkom poprzez uczynienie głosów niepoprawnymi, tajne głosowanie przez zewnętrznych gospodarzy byłoby również niewykrywalne. jedynym podejściem, które omija ten problem wydaje się... weryfikacja osobista. na przykład można mieć ekosystem „emitentów", w którym każdy emitent wydaje karty elektroniczne z kluczami prywatnymi, które użytkownik może natychmiast pobrać na swój smartfon i wysłać wiadomość, aby zastąpić klucz innym kluczem, którego nikomu nie ujawni. emitentami tymi mogą być spotkania i konferencje lub potencjalnie osoby, które przez mechanizm głosowania zostały już uznane za godne zaufania. budowanie infrastruktury umożliwiającej tworzenie mechanizmów odpornych na zmowy, w tym solidnych, zdecentralizowanych systemów tożsamości jest trudnym wyzwaniem. ale jeśli chcemy uwolnić potencjał takich mechanizmów, wydaje się nieuniknione, że musimy zrobić wszystko, co w naszej mocy, aby spróbować. prawdą jest, że obecny dogmat dotyczący bezpieczeństwa komputerowego, na przykład wprowadzenia głosowania online, to po prostu „nie rób tego". jeśli jednak chcemy rozszerzyć rolę mechanizmów podobnych do głosowania, w tym bardziej zaawansowanych form, takich jak głosowanie kwadratowe i finansowanie kwadratowe, do większej liczby zadań, to nie mamy innego wyjścia, jak stawić czoła wyzwaniu, zdecydowanie spróbować i mieć nadzieję, że uda nam się zrobić coś wystarczająco bezpiecznego, przynajmniej w niektórych zastosowaniach. dark mode toggle review of gitcoin quadratic funding round 3 2019 oct 24 see all posts special thanks to the gitcoin team and especially frank chen for working with me through these numbers the next round of gitcoin grants quadratic funding has just finished, and we the numbers for how much each project has received were just released. here are the top ten: altogether, $163,279 was donated to 80 projects by 477 contributors, augmented by a matching pool of $100,000. nearly half came from four contributions above $10,000: $37,500 to lighthouse, and $12,500 each to gas station network, black girls code and public health incentives layer. out of the remainder, about half came from contributions between $1,000 and $10,000, and the rest came from smaller donations of various sizes. but what matters more here are not the raw donations, but rather the subsidies that the quadratic funding mechanism applied. gitcoin grants is there to support valuable public goods in the ethereum ecosystem, but also serve as a testbed for this new quadratic donation matching mechanism, and see how well it lives up to its promise of creating a democratic, market-based and efficient way of funding public goods. this time around, a modified formula based on pairwise-bounded coordination subsidies was used, which has the goal of minimizing distortion from large contributions from coordinated actors. and now we get to see how the experiment went. judging the outcomes first, the results. ultimately, every mechanism for allocating resources, whether centralized, market-based, democratic or otherwise, must stand the test of delivering results, or else sooner or later it will be abandoned for another mechanism that is perceived to be better, even if it is less philosophically clean. judging results is inherently a subjective exercise; any single person's analysis of a mechanism will inevitably be shaped by how well the results fit their own preferences and tastes. however, in those cases where a mechanism does output a surprising result, one can and should use that as an opportunity to learn, and see whether or not one missed some key information that other participants in the mechanism had. in my own case, i found the top results very agreeable and a quite reasonable catalogue of projects that are good for the ethereum community. one of the disparities between these grants and the ethereum foundation grants is that the ethereum foundation grants (see recent rounds here and here) tend to overwhelmingly focus on technology with only a small section on education and community resources, whereas in the gitcoin grants while technology still dominates, ethhub is #2 and lower down defiprime.com is #14 and cryptoeconomics.study is #17. in this case my personal opinion is that ef has made a genuine error in undervaluing grants to community/education organizations and gitcoin's "collective instinct" is correct. score one for new-age fancy quadratic market democracy. another surprising result to me was austin griffith getting second place. i personally have never spent too much time thinking about burner wallet; i knew that it existed but in my mental space i did not take it too seriously, focusing instead on client development, l2 scaling, privacy and to a lesser extent smart contract wallets (the latter being a key use case of gas station network at #8). after seeing austin's impressive performance in this gitcoin round, i asked a few people what was going on. burner wallet (website, explainer article) is an "insta-wallet" that's very easy to use: just load it up on your desktop or phone, and there you have it. it was used successfully at ethdenver to sell food from food trucks, and generally many people appreciate its convenience. its main weaknesses are lower security and that one of its features, support for xdai, is dependent on a permissioned chain. austin's gitcoin grant is there to fund his ongoing work, and i have heard one criticism: there's many prototypes, but comparatively few "things taken to completion". there is also the critique that as great as austin is, it's difficult to argue that he's as important to the success of ethereum as, say, lighthouse and prysmatic, though one can reply that what matters is not total value, but rather the marginal value of giving a given project or person an extra $10,000. on the whole, however, i feel like quadratic funding's (glen would say deliberate!) tendency to select for things like burner wallet with populist appeal is a much needed corrective to the influence of the ethereum tech elite (including myself!) who often value technical impressiveness and undervalue simple and quick things that make it really easy for people to participate in ethereum. this one is slightly more ambiguous, but i'll say score two for new-age fancy quadratic market democracy. the main thing that i was disappointed the gitcoiner-ati did not support more was gitcoin maintenance itself. the gitcoin sustainability fund only got a total $1,119 in raw contributions from 18 participants, plus a match of $202. the optional 5% tips that users could give to gitcoin upon donating were not included into the quadratic matching calculations, but raised another ~$1,000. given the amount of effort the gitcoin people put in to making quadratic funding possible, this is not nearly enough; gitcoin clearly deserves more than 0.9% of the total donations in the round. meanwhile, the ethereum foundation (as well as consensys and individual donors) have been giving grants to gitcoin that include supporting gitcoin itself. hopefully in future rounds people will support gitcoin itself too, but for now, score one for good old-fashioned ef technocracy. on the whole, quadratic funding, while still young and immature, seems to be a remarkably effective complement to the funding preferences of existing institutions, and it seems worthwhile to continue it and even increase its scope and size in the future. pairwise-bounded quadratic funding vs traditional quadratic funding round 3 differs from previous rounds in that it uses a new flavor of quadratic funding, which limits the subsidy per pair of participants. for example, in traditional qf, if two people each donate $10, the subsidy would be $10, and if two people each donate $10,000, the subsidy would be $10,000. this property of traditional qf makes it highly vulnerable to collusion: two key employees of a project (or even two fake accounts owned by the same person) could each donate as much money as they have, and get back a very large subsidy. pairwise-bounded qf computes the total subsidy to a project by looking through all pairs of contributors, and imposes a maximum bound on the total subsidy that any given pair of participants can trigger (combined across all projects). pairwise-bounded qf also has the property that it generally penalizes projects that are dominated by large contributors: the projects that lost the most relative to traditional qf seem to be projects that have a single large contribution (or sometimes two). for example, "fuzz geth and parity for evm consensus bugs" got a $415 match compared to the $2000 he would have gotten in traditional qf; the decrease is explained by the fact that the contributions are dominated by two large $4500 contributions. on the other hand, cryptoeconomics.study got $1274, up nearly double from the $750 it would have gotten in traditional qf; this is explained by the large diversity of contributions that the project received and particularly the lack of large sponsors: the largest contribution to cryptoeconomics.study was $100. another desirable property of pairwise-bounded qf is that it privileges cross-tribal projects. that is, if there are projects that group a typically supports, and projects that group b typically supports, then projects that manage to get support from both groups get a more favorable subsidy (because the pairs that go between groups are not as saturated). has this incentive for building bridges appeared in these results? unfortunately, my code of honor as a social scientist obliges me to report the negative result: the ethereum community just does not yet have enough internal tribal structure for effects like this to materialize, and even when there are differences in correlations they don't seem strongly connected to higher subsidies due to pairwise-bounding. here are the cross-correlations between who contributed to different projects: generally, all projects are slightly positively correlated with each other, with a few exceptions with greater correlation and one exception with broad roughly zero correlation: nori (120 in this chart). however, nori did not do well in pairwise-bounded qf, because over 94% of its donations came from a single $5000 donation. dominance of large projects one other pattern that we saw in this round is that popular projects got disproportionately large grants: to be clear, this is not just saying "more contributions, more match", it's saying "more contributions, more match per dollar contributed". arguably, this is an intended feature of the mechanism. projects that can get more people to donate to them represent public goods that serve a larger public, and so tragedy of the commons problems are more severe and hence contributions to them should be multiplied more to compensate. however, looking at the list, it's hard to argue that, say, prysm ($3,848 contributed, $8,566 matched) is a more public good than nimbus ($1,129 contributed, $496 matched; for the unaware, prysm and nimbus are both eth2 clients). the failure does not look too severe; on average, projects near the top do seem to serve a larger public and projects near the bottom do seem niche, but it seems clear that at least part of the disparity is not genuine publicness of the good, but rather inequality of attention. n units of marketing effort can attract attention of n people, and theoretically get n^2 resources. of course, this could be solved via a "layer on top" venture-capital style: upstart new projects could get investors to support them, in return for a share of matched contributions received when they get large. something like this would be needed eventually; predicting future public goods is as important a social function as predicting future private goods. but we could also consider less winner-take-all alternatives; the simplest one would be adjusting the qf formula so it uses an exponent of eg. 1.5 instead of 2. i can see it being worthwhile to try a future round of gitcoin grants with such a formula (\(\left(\sum_i x_i^{\frac{2}{3}}\right)^{\frac{3}{2}}\) instead of \(\left(\sum_i x_i^{\frac{1}{2}}\right)^2\)) to see what the results are like. individual leverage curves one key question is, if you donate $1, or $5, or $100, how big an impact can you have on the amount of money that a project gets? fortunately, we can use the data to calculate these deltas! the different lines are for different projects; supporting projects with higher existing support will lead to you getting a bigger multiplier. in all cases, the first dollar is very valuable, with a matching ratio in some cases over 100:1. but the second dollar is much less valuable, and matching ratios quickly taper off; even for the largest projects increasing one's donation from $32 to $64 will only get a 1:1 match, and anything above $100 becomes almost a straight donation with nearly no matching. however, given that it's likely possible to get legitimate-looking github accounts on the grey market for around those costs, having a cap of a few hundred dollars on the amount of matched funds that any particular account can direct seems like a very reasonable mitigation, despite its costs in limiting the bulk of the matching effect to small-sized donations. conclusions on the whole, this was by far the largest and the most data-rich gitcoin funding round to date. it successfully attracted hundreds of contributors, reaching a size where we can finally see many significant effects in play and drown out the effects of the more naive forms of small-scale collusion. the experiment already seems to be leading to valuable information that can be used by future quadratic funding implementers to improve their quadratic funding implementations. the case of austin griffith is also interesting because $23,911 in funds that he received comes, in relative terms, surprisingly close to an average salary for a developer if the grants can be repeated on a regular schedule. what this means is that if gitcoin grants does continue operating regularly, and attracts and expands its pool of donations, we could be very close to seeing the first "quadratic freelancer" someone directly "working for the public", funded by donations boosted by quadratic matching subsidies. and at that point we could start to see more experimentation in new forms of organization that live on top of quadratic funding gadgets as a base layer. all in all, this foretells an exciting and, err, radical public-goods funding future ahead of us. dark mode toggle starks, part i: proofs with polynomials 2017 nov 09 see all posts special thanks to eli ben-sasson for ongoing help, explanations and review, coming up with some of the examples used in this post, and most crucially of all inventing a lot of this stuff; thanks to hsiao-wei wang for reviewing hopefully many people by now have heard of zk-snarks, the general-purpose succinct zero knowledge proof technology that can be used for all sorts of usecases ranging from verifiable computation to privacy-preserving cryptocurrency. what you might not know is that zk-snarks have a newer, shinier cousin: zk-starks. with the t standing for "transparent", zk-starks resolve one of the primary weaknesses of zk-snarks, its reliance on a "trusted setup". they also come with much simpler cryptographic assumptions, avoiding the need for elliptic curves, pairings and the knowledge-of-exponent assumption and instead relying purely on hashes and information theory; this also means that they are secure even against attackers with quantum computers. however, this comes at a cost: the size of a proof goes up from 288 bytes to a few hundred kilobytes. sometimes the cost will not be worth it, but at other times, particularly in the context of public blockchain applications where the need for trust minimization is high, it may well be. and if elliptic curves break or quantum computers do come around, it definitely will be. so how does this other kind of zero knowledge proof work? first of all, let us review what a general-purpose succinct zkp does. suppose that you have a (public) function \(f\), a (private) input \(x\) and a (public) output \(y\). you want to prove that you know an \(x\) such that \(f(x) = y\), without revealing what \(x\) is. furthermore, for the proof to be succinct, you want it to be verifiable much more quickly than computing \(f\) itself. let's go through a few examples: \(f\) is a computation that takes two weeks to run on a regular computer, but two hours on a data center. you send the data center the computation (ie. the code to run \(f\) ), the data center runs it, and gives back the answer \(y\) with a proof. you verify the proof in a few milliseconds, and are convinced that \(y\) actually is the answer. you have an encrypted transaction, of the form "\(x_1\) was my old balance. \(x_2\) was your old balance. \(x_3\) is my new balance. \(x_4\) is your new balance". you want to create a proof that this transaction is valid (specifically, old and new balances are non-negative, and the decrease in my balance cancels out the increase in your balance). \(x\) can be the pair of encryption keys, and \(f\) can be a function which contains as a built-in public input the transaction, takes as input the keys, decrypts the transaction, performs the check, and returns 1 if it passes and 0 if it does not. \(y\) would of course be 1. you have a blockchain like ethereum, and you download the most recent block. you want a proof that this block is valid, and that this block is at the tip of a chain where every block in the chain is valid. you ask an existing full node to provide such a proof. \(x\) is the entire blockchain (yes, all ?? gigabytes of it), \(f\) is a function that processes it block by block, verifies the validity and outputs the hash of the last block, and \(y\) is the hash of the block you just downloaded. so what's so hard about all this? as it turns out, the zero knowledge (ie. privacy) guarantee is (relatively!) easy to provide; there are a bunch of ways to convert any computation into an instance of something like the three color graph problem, where a three-coloring of the graph corresponds to a solution of the original problem, and then use a traditional zero knowledge proof protocol to prove that you have a valid graph coloring without revealing what it is. this excellent post by matthew green from 2014 describes this in some detail. the much harder thing to provide is succinctness. intuitively speaking, proving things about computation succinctly is hard because computation is incredibly fragile. if you have a long and complex computation, and you as an evil genie have the ability to flip a 0 to a 1 anywhere in the middle of the computation, then in many cases even one flipped bit will be enough to make the computation give a completely different result. hence, it's hard to see how you can do something like randomly sampling a computation trace in order to gauge its correctness, as it's just too easy to miss that "one evil bit". however, with some fancy math, it turns out that you can. the general very high level intuition is that the protocols that accomplish this use similar math to what is used in erasure coding, which is frequently used to make data fault-tolerant. if you have a piece of data, and you encode the data as a line, then you can pick out four points on the line. any two of those four points are enough to reconstruct the original line, and therefore also give you the other two points. furthermore, if you make even the slightest change to the data, then it is guaranteed at least three of those four points. you can also encode the data as a degree-1,000,000 polynomial, and pick out 2,000,000 points on the polynomial; any 1,000,001 of those points will recover the original data and therefore the other points, and any deviation in the original data will change at least 1,000,000 points. the algorithms shown here will make heavy use of polynomials in this way for error amplification. changing even one point in the original data will lead to large changes in a polynomial's trajectory a somewhat simple example suppose that you want to prove that you have a polynomial \(p\) such that \(p(x)\) is an integer with \(0 \leq p(x) \leq 9\) for all \(x\) from 1 to 1 million. this is a simple instance of the fairly common task of "range checking"; you might imagine this kind of check being used to verify, for example, that a set of account balances is still positive after applying some set of transactions. if it were \(1 \leq p(x) \leq 9\), this could be part of checking that the values form a correct sudoku solution. the "traditional" way to prove this would be to just show all 1,000,000 points, and verify it by checking the values. however, we want to see if we can make a proof that can be verified in less than 1,000,000 steps. simply randomly checking evaluations of \(p\) won't do; there's always the possibility that a malicious prover came up with a \(p\) which satisfies the constraint in 999,999 places but does not satisfy it in the last one, and random sampling only a few values will almost always miss that value. so what can we do? let's mathematically transform the problem somewhat. let \(c(x)\) be a constraint checking polynomial; \(c(x) = 0\) if \(0 \leq x \leq 9\) and is nonzero otherwise. there's a simple way to construct \(c(x)\): \(x \cdot (x-1) \cdot (x-2) \cdot \ldots(x-9)\) (we'll assume all of our polynomials and other values use exclusively integers, so we don't need to worry about numbers in between). now, the problem becomes: prove that you know \(p\) such that \(c(p(x)) = 0\) for all \(x\) from 1 to 1,000,000. let \(z(x) = (x-1) \cdot (x-2) \cdot \ldots (x-1000000)\). it's a known mathematical fact that any polynomial which equals zero at all \(x\) from 1 to 1,000,000 is a multiple of \(z(x)\). hence, the problem can now be transformed again: prove that you know \(p\) and \(d\) such that \(c(p(x)) = z(x) \cdot d(x)\) for all \(x\) (note that if you know a suitable \(c(p(x))\) then dividing it by \(z(x)\) to compute \(d(x)\) is not too difficult; you can use long polynomial division or more realistically a faster algorithm based on ffts). now, we've converted our original statement into something that looks mathematically clean and possibly quite provable. so how does one prove this claim? we can imagine the proof process as a three-step communication between a prover and a verifier: the prover sends some information, then the verifier sends some requests, then the prover sends some more information. first, the prover commits to (ie. makes a merkle tree and sends the verifier the root hash of) the evaluations of \(p(x)\) and \(d(x)\) for all \(x\) from 1 to 1 billion (yes, billion). this includes the 1 million points where \(0 \leq p(x) \leq 9\) as well as the 999 million points where that (probably) is not the case. we assume the verifier already knows the evaluation of \(z(x)\) at all of these points; the \(z(x)\) is like a "public verification key" for this scheme that everyone must know ahead of time (clients that do not have the space to store \(z(x)\) in its entirety can simply store the merkle root of \(z(x)\) and require the prover to also provide branches for every \(z(x)\) value that the verifier needs to query; alternatively, there are some number fields over which \(z(x)\) for certain \(x\) is very easy to calculate). after receiving the commitment (ie. merkle root) the verifier then selects a random 16 \(x\) values between 1 and 1 billion, and asks the prover to provide the merkle branches for \(p(x)\) and \(d(x)\) there. the prover provides these values, and the verifier checks that (i) the branches match the merkle root that was provided earlier, and (ii) \(c(p(x))\) actually equals \(z(x) \cdot d(x)\) in all 16 cases. we know that this proof perfect completeness if you actually know a suitable \(p(x)\), then if you calculate \(d(x)\) and construct the proof correctly it will always pass all 16 checks. but what about soundness that is, if a malicious prover provides a bad \(p(x)\), what is the minimum probability that they will get caught? we can analyze as follows. because \(c(p(x))\) is a degree-10 polynomial composed with a degree-1,000,000 polynomial, its degree will be at most 10,000,000. in general, we know that two different degree-\(n\) polynomials agree on at most \(n\) points; hence, a degree-10,000,000 polynomial which is not equal to any polynomial which always equals \(z(x) \cdot d(x)\) for some \(x\) will necessarily disagree with them all at at least 990,000,000 points. hence, the probability that a bad \(p(x)\) will get caught in even one round is already 99%; with 16 checks, the probability of getting caught goes up to \(1 10^{-32}\); that is to say, the scheme is about as hard to spoof as it is to compute a hash collision. so... what did we just do? we used polynomials to "boost" the error in any bad solution, so that any incorrect solution to the original problem, which would have required a million checks to find directly, turns into a solution to the verification protocol that can get flagged as erroneous at 99% of the time with even a single check. we can convert this three-step mechanism into a non-interactive proof, which can be broadcasted by a single prover once and then verified by anyone, using the fiat-shamir heuristic. the prover first builds up a merkle tree of the \(p(x)\) and \(d(x)\) values, and computes the root hash of the tree. the root itself is then used as the source of entropy that determines what branches of the tree the prover needs to provide. the prover then broadcasts the merkle root and the branches together as the proof. the computation is all done on the prover side; the process of computing the merkle root from the data, and then using that to select the branches that get audited, effectively substitutes the need for an interactive verifier. the only thing a malicious prover without a valid \(p(x)\) can do is try to make a valid proof over and over again until eventually they get extremely lucky with the branches that a merkle root that they compute selects, but with a soundness of \(1 10^{-32}\) (ie. probability of at least \(1 10^{-32}\) that a given attempted fake proof will fail the check) it would take a malicious prover billions of years to make a passable proof. going further to illustrate the power of this technique, let's use it to do something a little less trivial: prove that you know the millionth fibonacci number. to accomplish this, we'll prove that you have knowledge of a polynomial which represents a computation tape, with \(p(x)\) representing the \(x\)th fibonacci number. the constraint checking polynomial will now hop across three x-coordinates: \(c(x_1, x_2, x_3) = x_3-x_2-x_1\) (notice how if \(c(p(x), p(x+1), p(x+2)) = 0\) for all \(x\) then \(p(x)\) represents a fibonacci sequence). the translated problem becomes: prove that you know \(p\) and \(d\) such that \(c(p(x), p(x+1), p(x+2)) = z(x) \cdot d(x)\). for each of the 16 indices that the proof audits, the prover will need to provide merkle branches for \(p(x)\), \(p(x+1)\), \(p(x+2)\) and \(d(x)\). the prover will additionally need to provide merkle branches to show that \(p(0) = p(1) = 1\). otherwise, the entire process is the same. now, to accomplish this in reality there are two problems that need to be resolved. the first problem is that if we actually try to work with regular numbers the solution would not be efficient in practice, because the numbers themselves very easily get extremely large. the millionth fibonacci number, for example, has 208988 digits. if we actually want to achieve succinctness in practice, instead of doing these polynomials with regular numbers, we need to use finite fields number systems that still follow the same arithmetic laws we know and love, like \(a \cdot (b+c) = (a\cdot b) + (a\cdot c)\) and \((a^2 b^2) = (a-b) \cdot (a+b)\), but where each number is guaranteed to take up a constant amount of space. proving claims about the millionth fibonacci number would then require a more complicated design that implements big-number arithmetic on top of this finite field math. the simplest possible finite field is modular arithmetic; that is, replace every instance of \(a + b\) with \(a + b \mod{n}\) for some prime \(n\), do the same for subtraction and multiplication, and for division use modular inverses (eg. if \(n = 7\), then \(3 + 4 = 0\), \(2 + 6 = 1\), \(3 \cdot 4 = 5\), \(4 / 2 = 2\) and \(5 / 2 = 6\)). you can learn more about these kinds of number systems from my description on prime fields here (search "prime field" in the page) or this wikipedia article on modular arithmetic (the articles that you'll find by searching directly for "finite fields" and "prime fields" unfortunately tend to be very complicated and go straight into abstract algebra, don't bother with those). second, you might have noticed that in my above proof sketch for soundness i neglected to cover one kind of attack: what if, instead of a plausible degree-1,000,000 \(p(x)\) and degree-9,000,000 \(d(x)\), the attacker commits to some values that are not on any such relatively-low-degree polynomial? then, the argument that an invalid \(c(p(x))\) must differ from any valid \(c(p(x))\) on at least 990 million points does not apply, and so different and much more effective kinds of attacks are possible. for example, an attacker could generate a random value \(p\) for every \(x\), then compute \(d = c(p) / z(x)\) and commit to these values in place of \(p(x)\) and \(d(x)\). these values would not be on any kind of low-degree polynomial, but they would pass the test. it turns out that this possibility can be effectively defended against, though the tools for doing so are fairly complex, and so you can quite legitimately say that they make up the bulk of the mathematical innovation in starks. also, the solution has a limitation: you can weed out commitments to data that are very far from any degree-1,000,000 polynomial (eg. you would need to change 20% of all the values to make it a degree-1,000,000 polynomial), but you cannot weed out commitments to data that only differ from a polynomial in only one or two coordinates. hence, what these tools will provide is proof of proximity proof that most of the points on \(p\) and \(d\) correspond to the right kind of polynomial. as it turns out, this is sufficient to make a proof, though there are two "catches". first, the verifier needs to check a few more indices to make up for the additional room for error that this limitation introduces. second, if we are doing "boundary constraint checking" (eg. verifying \(p(0) = p(1) = 1\) in the fibonacci example above), then we need to extend the proof of proximity to not only prove that most points are on the same polynomial, but also prove that those two specific points (or whatever other number of specific points you want to check) are on that polynomial. in the next part of this series, i will describe the solution to proximity checking in much more detail, and in the third part i will describe how more complex constraint functions can be constructed to check not just fibonacci numbers and ranges, but also arbitrary computation. dark mode toggle na colusão 2000 jan 01 see all posts agradecimentos especiais a glen weyl, phil daian e jinglan wang para revisão, e jeff prestes para tradução nos últimos anos, tem havido um interesse crescente em usar incentivos econômicos e design de mecanismos para alinhar o comportamento dos participantes em vários contextos. no mundo blockchain, o design dos mecanismos fornece, acima de tudo, a segurança para o próprio blockchain incentivando mineradores ou validadores de provas de depósitos com desafios de modo a que eles participem honestamente, mas, mais recentemente, está sendo também aplicado em mercados de predição, "token com curadoria" e outros contextos. o nascente movimento radicalxchange entretanto fez surgir experimentações com harberger taxes, voto quadrático, financiamento quadrático e muito mais. mais recentemente, tem havido também um interesse crescente em utilizar incentivos baseados em tokens para tentar encorajar textos de qualidade nos meios de comunicação social. no entanto, à medida que o desenvolvimento destes sistemas se aproxima da teoria à prática há uma série de desafios que têm de ser enfrentados, desafios que, a meu ver, ainda não foram adequadamente enfrentados. como um exemplo recente deste movimento da teoria para a implantação, surgiu a bihu, uma plataforma chinesa que libertou recentemente um mecanismo baseado em moedas para incentivar as pessoas a escrever postagens. o mecanismo básico (veja o whitepaper em chinês aqui) é que se um usuário da plataforma possui tokens key, ele têm a capacidade de colocar esses tokens key nos artigos; todos os usuários podem fazer k "votos favoritos" por dia, e o "peso" de cada voto positivo é proporcional à participação do usuário fazendo o voto favorável. artigos com uma maior quantidade de participações votando-os são exibids de maneira mais proeminente, e o autor do artigo recebe uma recompensa de key tokens aproximadamente proporcional à quantidade de votos positivos do key. trata-se de uma simplificação excessiva e o próprio mecanismo tem nele algumas não-linearidades em si mas não são essenciais para o funcionamento básico do mecanismo. key tem valor porque pode ser usado de várias formas dentro da plataforma, mas especialmente uma porcentagem de todas as receitas de anúncios é usada para comprar e queimar key (ótimo!, grande ponto positivo a eles por fazer isto e não fazer outro token de meio de troca!). este tipo de design está longe de ser único. incentivar a criação de conteúdo online é algo com que muitas pessoas se preocupam e houve muitas tentativas de natureza semelhante, bem como alguns com desenhos bastante diferentes. e, neste caso, esta plataforma específica já está sendo utilizada de forma significativa: alguns meses atrás, o subreddit sobre negociações na ethereum /r/ethtrader introduziu um recurso experimental pouco similar onde é emitido um token chamado "donuts" para usuários que fazem comentários que recebem um voto favorito, com uma quantidade definida de doações semanais para os usuários, proporcionais ao número de votos positivos que seus comentários receberam. os donuts poderiam ser usados para comprar o direito de definir o conteúdo do banner no topo do subreddit, e poderiam também ser utilizados para votar nas sondagens comunitárias. no entanto, ao contrário do que acontece no sistema key, aqui as recompensas que b recebe quando a vota favoravelmente a b não são proporcionais à oferta existente de moeda de a. em vez disso, cada conta reddit tem a mesma capacidade de contribuir para outras contas do reddit. este tipo de experimentos, que tentam recompensar a criação de conteúdo de qualidade de uma forma que vá além das limitações conhecidas de doações/micropublicações, são muito valiosos; subremuneração do conteúdo de internet gerado pelo usuário é um problema muito significativo na sociedade em geral (veja "radicalismo liberal" e "dados como trabalho"), e é animador ver comunidades criptográficas tentando usar o poder do design do mecanismo para fazer incursões na resolução do problema. mas, infelizmente, estes sistemas também são vulneráveis ao ataque. votação independente, plutocracia e subornos aqui está como se pode atacar economicamente o projeto proposto acima. suponha que algum usuário rico adquira alguma quantidade n de tokens, e como resultado, cada um dos k de votos a favor dos usuários dá ao destinatário uma recompensa de n * q (q aqui provavelmente sendo um número muito pequeno, p. ex. pensar q = 0.000001). o usuário simplesmente vota a favor das suas próprias contas de sockpuppet, dando a si mesmo a recompensa de n * k * q. então, o sistema simplesmente entra em colapso com cada usuário com uma "taxa de juro" de k * q por período, e o mecanismo não realiza mais nada. o atual mecanismo do bihu parecia prever isto, e tem alguma lógica superlinear em que artigos com mais key positivos ganham uma recompensa desproporcionadamente maior, aparentemente para encorajar votos em publicações populares ao invés de se auto-votar. acrescentar este tipo de linearidade, para evitar que o auto-voto prejudique todo o sistema, é um padrão comum entre os sistemas de governança de votos; a maioria dos regimes dpos tem um número limitado de faixas horárias delegadas com zero recompensas para quem não tiver votos suficientes para se juntar a um dos slots, com efeito similar. mas estes regimes introduzem invariavelmente dois novos pontos fracos: eles subsidiam plutocracia , já que indivíduos muito ricos e cartéis ainda podem obter fundos suficientes para votar em si. eles podem ser contornados pelos usuários subornando outros usuários para votar neles em massa. ataques de suborno podem parecer distantes (quem aqui já aceitou um suborno na vida real?!), mas num ecossistema maduro isso é muito mais realista do que parece. na maioria dos contextos onde o suborno ocorreu no mundo blockchain, os operadores usam um novo nome eufemista para dar ao conceito um rosto amigável: não é um suborno, é um "pool de depósitos" que "partilha dividendos". subornos podem até ser ofuscados: imaginem uma bolsa de criptomoedas que oferece taxas zero e gasta o esforço para fazer uma interface de usuário anormalmente boa e nem sequer tenta obter lucros. em vez disso, utiliza as moedas que os clientes depositam para participar em vários sistemas de votação de moedas. também haverá, inevitavelmente, pessoas que vêem a cumplicidade no grupo como simplesmente normal; veja um recente escândalo envolvendo eos dpos por exemplo: por último, existe a possibilidade de um "suborno negativo", chantagem ou coação, ameaçando os participantes com danos, a menos que atuem dentro do mecanismo de uma certa maneira. no experimento /r/ethtrader, o medo de que as pessoas entrem e comprem donuts para mudar as sondagens de governança levaram a comunidade a decidir fazer apenas trancado (ex. não negociáveis) donuts elegíveis para uso na votação. mas há um ataque ainda mais barato do que comprar donuts (um ataque que pode ser considerado como um tipo de suborno ofuscado): alugando eles. se um invasor já estiver segurando eth, eles podem usá-lo como garantia em uma plataforma como compound para contrair um empréstimo de algum token, dando a você todo o direito de usar esse token para qualquer fim, incluindo participar de votos, e quando eles estão feitos, eles simplesmente enviam os tokens de volta ao contrato de empréstimo para obter suas garantias de volta tudo sem terem que suportar nem um segundo de exposição de preços para o token que eles usavam apenas para balançar um voto de moeda, mesmo que o mecanismo de voto em moedas inclua um tempo de bloqueio (por exemplo o que o bihu faz). em todos os casos, as questões relacionadas com o suborno e, acidentalmente, com o excesso de poder dos participantes bem conectados e ricos, são surpreendentemente difíceis de evitar. identidade alguns sistemas tentam mitigar os aspectos plutocráticos dos votos das moedas, fazendo uso de um sistema de identidade. no caso do sistema de donut /r/ethtrader, por exemplo, embora sondagens de governança sejam feitas via votação de moedas, o mecanismo que determina quantos donuts</0> (no caso moedas) você recebe no primeiro lugar é baseado em contas do reddit: 1 voto a favor de 1 conta reddit = n donuts ganhados. o objetivo ideal de um sistema de identidade é tornar relativamente fácil para os indivíduos obter uma identidade, mas relativamente difícil de conseguir muitas identidades. no sistema de donut /r/ethtrader, isso é contas reddit, no gadget de verificação gitcoin clr são contas do github que são usadas para o mesmo propósito. mas a identidade, pelo menos na forma como tem sido aplicada até agora, é uma coisa frágil.... ah, você é muito preguiçoso para fazer um grande racha de telefones? bem, talvez você esteja procurando por isto: normalmente avisos sobre como sites esboçados podem ou não enganar você, realizar suas próprias pesquisas, etc. etc. se aplica. provavelmente, atacar esses mecanismos simplesmente controlando milhares de identidades falsas como um fantoche é ainda mais fácil do que ter que brigar pessoas. e se você acha que a resposta é apenas aumentar a segurança para subir até ids emitidos a nível governamental? bem, se você quer alguns deles, você pode começar a explorar aqui, mas tenha em mente que existem organizações criminosas especializadas que estão muito à frente de você e mesmo que todos as organizações nefastas sejam retiradas, governos hostis definitivamente vão criar passaportes falsificados pelos milhões, se somos estúpidos o suficiente para criar sistemas que tornam esse tipo de atividade rentável. e isto nem sequer começa a mencionar ataques no sentido oposto, instituições emissoras de identidades tentando desempoderar comunidades marginalizadas negando a elas documentos de identidade... colusão dado que tantos mecanismos parecem falhar de formas tão parecidas, assim que múltiplas identidades ou mesmo mercados líquidos entram na situação, podemos perguntar: existe alguma profunda vertente comum que cause todas estas questões? diria que a resposta é sim, e o "fio comum" é este: é muito mais difícil e muito mais impossível, criar mecanismos que mantenham propriedades desejáveis em um modelo onde os participantes podem colidir, do que em um modelo onde não podem. a maioria das pessoas provavelmente já tem alguma intuição sobre isso; exemplos específicos deste princípio estão por trás de normas bem estabelecidas e, frequentemente, de leis que promovem mercados competitivos e restringem os cartéis fixos de preços, a compra e venda de votos, e o suborno. mas a questão é muito mais profunda e geral. na versão da teoria do jogo que se concentra na escolha individual isto é, a versão que parte do princípio de que cada participante toma decisões de forma independente e que não permite a possibilidade de grupos de agentes trabalharem como uma única para seu benefício mútuo, existem provas matemáticas que pelo menos um equilíbrio de nash estável deve existir em qualquer jogo, e designers de mecanismos têm uma latitude muito ampla para "desenvolver" jogos para obter resultados específicos. mas na versão da teoria do jogo que permite a possibilidade de coligações trabalhando juntos, chamada teoria cooperativa do jogo, existem classes grandes de jogos que não têm nenhum resultado estável que uma coligação não possa desviar de forma rentável. jogos majoritários, formalmente descrito como jogos de agentes n onde qualquer subconjunto de mais de metade deles pode capturar uma recompensa fixa e dividi-la entre si, uma configuração semelhante a muitas situações no governo corporativo, a política e muitas outras situações na vida humana, são parte desse conjunto de jogos inerentemente instáveis. ou seja, se existe uma situação com um conjunto fixo de recursos e um mecanismo atualmente estabelecido para a distribuição desses recursos e é inevitavelmente possível para 51% dos participantes poderem conspirar para tomar controle dos recursos não importa qual é a configuração atual, há sempre alguma conspiração que pode emergir que seria lucrativa para os participantes. no entanto, essa conspiração seria, por sua vez, vulnerável a potenciais novas conspirações, incluindo possivelmente uma combinação de conspiradores anteriores e vítimas... e assim por diante. rodada a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 este fato, a instabilidade dos jogos majoritários na teoria do jogo cooperativo, é indiscutivelmente altamente subavaliado como um modelo matemático geral simplificado que permite que não exista um "fim da história" em política e um sistema que se revele totalmente satisfatório. pessoalmente, acredito que é muito mais útil do que a famosa teorema da seta , por exemplo. há duas maneiras de contornar esta questão. o primeiro é tentar restringir-nos à classe de jogos que são "sem identidade" e "sem conluio seguro", portanto, nos casos em que não precisamos de nos preocupar com subornos ou identidades. o segundo é tentar atacar directamente os problemas de identidade e de resistência de conluio e na verdade resolvê-los bem o suficiente para que possamos implementar jogos não-seguros com as propriedades mais ricas que eles oferecem. **desenho de jogos sem identidade e sem conluio a classe de jogos que é livre de identidade e segura de conluio é substancial. até mesmo a prova de trabalho é segura até o limite de um único ator ter ~23.1% do total de hashpower, e este limite pode ser aumentado até 50% com engenharia inteligente. os mercados competitivos estão razoavelmente seguros até um limite relativamente elevado, o que é facilmente alcançado em alguns casos, mas em outros não o é. no caso de governança e da curadoria de conteúdo (ambos são realmente apenas casos especiais do problema geral de identificação de bens públicos e emblemas públicos) uma grande classe de mecanismo que funciona bem é futarchy normalmente retratada como "governança por mercado de previsão", embora eu também defenda que a utilização de depósitos de segurança se situa fundamentalmente na mesma classe de técnica. a forma como os mecanismos de contestação, na sua forma mais geral, funcionam é que eles fazem "votar" e não apenas uma expressão de opinião, mas também uma previsão , com uma recompensa por fazer previsões que são verdadeiras e uma penalização por fazer previsões que são falsas. por exemplo, minha proposta para "mercados de predição de daos de curadoria de conteúdo" sugere um design semi-centralizado onde qualquer pessoa pode votar a favor ou contra o conteúdo enviado com conteúdos que são mais visíveis, onde existe também um "painel de moderação" que toma as decisões finais. para cada postagem, há uma pequena probabilidade (proporcional ao volume total de votos+votos contra naquele post) de o painel de moderação ser chamado a tomar uma decisão final sobre o post. se o painel de moderação aprovar uma postagem, todos os que votaram a favor são recompensados e todos os que votaram contra são penalizados, e se o painel de moderação desaprova um post, acontece o contrário; este mecanismo encoraja os participantes a compor votos positivos e votos negativos que tentam "prever" os acórdãos do painel de moderação. outro possível exemplo de negação é um sistema de governança para um projeto com um token, onde quem vota a favor de uma decisão é obrigado a adquirir uma quantidade de fichas ao preço no momento em que a votação se inicia se a votação tiver resultado; isto garante que a votação de uma má decisão é cara, e no limite, no caso de uma má decisão resultar numa votação, todos os que aprovaram a decisão terão de vender a sua mercadoria essencialmente a todos os que participam no projeto. isto garante que um voto individual a favor de uma decisão "errada" possa ser muito dispendioso para o eleitor, excluindo a possibilidade de ataques de suborno barato. uma descrição gráfica de uma forma de contestação, a criação de dois mercados que representam os dois "possíveis futuros mundos" e a escolha do mercado com um preço mais favorável. fonte este post em ethresear.ch no entanto, esse leque de coisas que mecanismos deste tipo podem fazer é limitado. no caso do exemplo de curadoria de conteúdo acima, não estamos realmente resolvendo a governança, estamos apenas dimensionando a funcionalidade de um gadget de governança que já é considerado confiável. poderia se tentar substituir o painel de moderação por um mercado de predição sobre o preço de um token que representa o direito de comprar espaço publicitário, mas, na prática, os preços são um indicador demasiado ruidoso para que tudo isto seja viável para além de um número muito reduzido de decisões muito grandes. e muitas vezes o valor que estamos tentando maximizar é explicitamente algo diferente do valor máximo de uma moeda. vamos olhar mais claramente porquê. no caso mais geral, onde não podemos facilmente determinar o valor de uma decisão de governança através do seu impacto no preço de um token, bons mecanismos para identificar bens e maus públicos, infelizmente, não podem ser isentos de identidade, nem ser seguros para conluios. se tentarmos preservar a propriedade de um jogo sem identidade, construindo um sistema onde as identidades não importam e apenas as moedas importam existe um compromisso impossível entre falhar em incentivar bens públicos legítimos ou subsidiar excessivamente a plutocracia. o argumento é o seguinte. suponha que há algum autor que esteja produzindo um bem público (por exemplo uma série de postagens do blog) que fornece valor para cada membro de uma comunidade de 10.000 pessoas. suponha que existe algum mecanismo onde os membros da comunidade podem tomar uma ação que faz com que o autor receba um ganho de us$ 1. a menos que os membros da comunidade sejam extremamente altruístas, o custo de executar esta ação deve ser muito menor do que us$ 1, como se a porção do benefício capturado pelo membro da comunidade que apoia o autor fosse muito menor que o custo de suporte ao autor, e assim o sistema cai em uma tragédia do comum onde ninguém apoia o autor. portanto, deve existir uma maneira de fazer com que o autor ganhe us$ 1 por um custo muito inferior a us$ 1 mas agora suponhamos que existe também uma comunidade falsa, que consiste em 10.000 contas falsas de um mesmo agressor abastado que é portador de riqueza. esta comunidade toma todas as mesmas ações que a comunidade real, exceto ao invés de apoiar o autor, eles suportam outra conta falsa que também é um fantoche do invasor. se foi possível para um membro da "comunidade real" dar ao autor \(1 a um custo pessoal muito inferior a us\) 1, é possível o invasor dar a ele mesmo us$ 1 a um custo muito menor que us$ 1 muitas vezes drenando assim o financiamento do sistema. qualquer mecanismo que possa ajudar partes verdadeiramente subcoordenadas a coordenar irá, sem as salvaguardas certas, também ajudar grupos já coordenados (como muitas contas controladas pela mesma pessoa) sobrecoordenado, extraindo dinheiro do sistema. um desafio semelhante surge quando o objetivo não é o financiamento, mas sim determinar qual o conteúdo que deve ser mais visível. qual o conteúdo que você acha que teria mais valor, em dólar, para suportá-lo: um artigo de blog de alta qualidade legitimamente vantajoso para milhares de pessoas, mas beneficiando cada pessoa de forma relativamente ligeira ou isso? ou talvez isso? aqueles que têm acompanhado políticas recentes "no mundo real" poderiam também apontar um tipo diferente de conteúdo que beneficia atores altamente centralizados: a manipulação das redes sociais por parte de governos hostis. em última análise, tanto os sistemas centralizados como os sistemas descentralizados enfrentam o mesmo problema fundamental que é: o mercado "das ideias" (e dos bens públicos em geral) está muito longe de um "mercado eficiente" no sentido em que os economistas normalmente usam o termo , e isto conduz à subprodução de bens públicos, mesmo em "tempo de paz", mas também à vulnerabilidade a ataques ativos. é só um problema difícil. é também por isso que os sistemas de votação baseados em moedas (como o bihu) têm uma grande vantagem genuína sobre os sistemas baseados na identidade (como o gitcoin clr ou a experiência de donut /r/ethtrader: pelo menos não há nenhum benefício em comprar contas em massa, porque tudo o que você faz é proporcional ao número de moedas que você tem, independentemente do número de contas que as moedas estão divididas entre si. no entanto, mecanismos que não dependem de qualquer modelo de identidade e apenas dependem de moedas fundamentalmente não conseguem resolver o problema de interesses concentrados que são concorrentes e que tentam apoiar os bens públicos; um mecanismo sem identidade que autoriza comunidades distribuídas não pode evitar empoderamento excessivo de plutocratas centralizados fingindo ser distribuídos por comunidades. mas não são apenas problemas de identidade que os jogos de bens públicos também são vulneráveis; também são subornos. para ver porquê, considere novamente o exemplo acima, mas onde a "comunidade falsa" é 10.001 sockpuppets do atacante, o atacante tem apenas uma identidade, a conta que recebe financiamento, e as outras 10.000 contas são usuários reais mas usuários que recebem um suborno de us$ 0,1 cada um para tomar a ação que faria o atacante ganhar mais us$ 1. como mencionado acima, estes subornos podem ser altamente ofuscados, mesmo através de serviços de custódia de terceiros que votam em nome do usuário em troca de conveniência, e no caso do "voto de moeda", um suborno ofuscado é ainda mais fácil: é possível alugar moedas no mercado e utilizá-las para participar em votações. assim, embora alguns tipos de jogos, especialmente o mercado de predição ou os jogos baseados em depósitos de segurança, podem se tornar seguros e isentos de identidade. o financiamento generalizado de bens públicos parece ser uma classe de problemas em que não é possível fazer funcionar abordagens seguras e sem identidade. resistência ao conluio e identidade a outra alternativa é atacar de frente o problema de identidade. como mencionado acima, simplesmente ir até sistemas centralizados de identidade de maior segurança, como passaportes e outras identificações do governo, não funcionará na escala; num contexto suficientemente incentivado, elas são muito inseguras e vulneráveis aos próprios governos emissores! ao invés o tipo de "identidade" de que estamos falando aqui é uma espécie de robusto conjunto multi fatorial de afirmações de que um ator identificado por algum conjunto de mensagens é, de fato, um indivíduo único. um recente proto-modelo deste tipo de identidade em rede é a recuperação social no smartphone do blockchain do htc: a ideia básica é que sua chave privada é compartilhada em segredo entre até cinco contatos confiáveis, de tal forma que matematicamente garante que três deles possam recuperar a chave original, mas dois ou menos não podem. isso se qualifica como um "sistema de identidade" são seus cinco amigos determinando se alguém que tenta recuperar sua conta é realmente você. no entanto, é um sistema de identidade especial tentando resolver um problema segurança da conta pessoal que é diferente (e mais fácil!) do problema de tentar identificar humanos únicos. dito isto, é muito possível que o modelo geral dos indivíduos que se identificam uns sobre os outros seja introduzido numa espécie de modelo de identidade mais robusto. estes sistemas podem ser ampliados, se desejado, usando o modelo "futarchy" descrito acima: se alguém fizer uma reivindicação de que alguém é um homem único, e alguém discorda, e ambos os lados estão dispostos a estabelecer uma ligação para contestar a questão, o sistema pode convocar um painel de decisão para determinar quem tem razão. mas queremos também uma outra propriedade de importância crucial: queremos uma identidade que não se pode alugar ou vender. claro, não podemos impedir que as pessoas façam um acordo "você me manda $50, eu te mandarei minha chave", mas o que nós podemos tentar fazer é impedir que tais negócios sejam rentáveis torná-los assim o vendedor pode facilmente enganar o comprador e dar ao comprador uma chave que não funciona realmente. uma maneira de fazer isso é fazer um mecanismo através do qual o proprietário de uma chave pode enviar uma transação que revoga a chave e a substitui por outra chave da escolha do proprietário, de uma forma que não pode ser provada. talvez a maneira mais simples de contornar isto seja usar uma parte confiável que administra o cálculo e publica apenas resultados (juntamente com zero provas de conhecimento provando os resultados, então a parte confiável é confiável apenas pela privacidade, não pela integridade), ou descentralizar a mesma funcionalidade através da computação multipartidaria. tais abordagens não resolverão completamente o conluio; um grupo de amigos ainda poderia se juntar e sentar-se no mesmo lugar e coordenar os votos, mas pelo menos o reduzirão a um nível exequível que não levará a que estes sistemas sofram uma completa falha. há ainda um outro problema: a distribuição inicial da chave. o que acontece se um usuário criar sua identidade dentro de um serviço de custódia de terceiros que depois armazena a chave privada e a usa para votar clandestinamente? este seria um suborno implícito, o poder de voto do usuário em troca de fornecer ao usuário um serviço conveniente, e mais, se o sistema estiver seguro na medida em que ele previne subornos tornando inprováveis as votações clandestinas de anfitriões de terceiros seriam também não detectáveis. a única abordagem que escamoteia este problema parece ser.... verificação em pessoa. por exemplo, poderíamos ter um ecossistema de "emitentes" onde cada um emite cartões inteligentes com chaves privadas, que o usuário pode baixar imediatamente para seu smartphone e enviar uma mensagem para substituir a chave por uma chave diferente que eles não revelam para ninguém. estes emitentes poderiam ser encontros e conferências, ou potencialmente indivíduos que já foram considerados confiáveis por algum mecanismo de votação. construir a infraestrutura para tornar possíveis mecanismos de resistência a conluios, incluindo robustos sistemas de identidade descentralizados é um desafio difícil, mas se queremos desbloquear o potencial de tais mecanismos, é inevitável que tenhamos de fazer o nosso melhor para tentar. é verdade que o atual dogma da segurança dos computadores, por exemplo, introduzir voto online é simplesmente um "não", mas se queremos expandir o papel dos mecanismos de votação incluindo formas mais avançadas, como votações quadráticas e finanças quadraticas, para mais funções, não temos escolha senão enfrentar o desafio frontalmente, tentar fazer algo suficientemente seguro e esperar que seja bem sucedido, pelo menos para alguns casos de uso. dark mode toggle [mirror] zk-snarks: under the hood 2017 feb 01 see all posts this is a mirror of the post at https://medium.com/@vitalikbuterin/zk-snarks-under-the-hood-b33151a013f6 this is the third part of a series of articles explaining how the technology behind zk-snarks works; the previous articles on quadratic arithmetic programs and elliptic curve pairings are required reading, and this article will assume knowledge of both concepts. basic knowledge of what zk-snarks are and what they do is also assumed. see also christian reitwiessner's article here for another technical introduction. in the previous articles, we introduced the quadratic arithmetic program, a way of representing any computational problem with a polynomial equation that is much more amenable to various forms of mathematical trickery. we also introduced elliptic curve pairings, which allow a very limited form of one-way homomorphic encryption that lets you do equality checking. now, we are going to start from where we left off, and use elliptic curve pairings, together with a few other mathematical tricks, in order to allow a prover to prove that they know a solution for a particular qap without revealing anything else about the actual solution. this article will focus on the pinocchio protocol by parno, gentry, howell and raykova from 2013 (often called pghr13); there are a few variations on the basic mechanism, so a zk-snark scheme implemented in practice may work slightly differently, but the basic principles will in general remain the same. to start off, let us go into the key cryptographic assumption underlying the security of the mechanism that we are going to use: the *knowledge-of-exponent* assumption. basically, if you get a pair of points \(p\) and \(q\), where \(p \cdot k = q\), and you get a point \(c\), then it is not possible to come up with \(c \cdot k\) unless \(c\) is "derived" from \(p\) in some way that you know. this may seem intuitively obvious, but this assumption actually cannot be derived from any other assumption (eg. discrete log hardness) that we usually use when proving security of elliptic curve-based protocols, and so zk-snarks do in fact rest on a somewhat shakier foundation than elliptic curve cryptography more generally — although it's still sturdy enough that most cryptographers are okay with it. now, let's go into how this can be used. supposed that a pair of points \((p, q)\) falls from the sky, where \(p \cdot k = q\), but nobody knows what the value of \(k\) is. now, suppose that i come up with a pair of points \((r, s)\) where \(r \cdot k = s\). then, the koe assumption implies that the only way i could have made that pair of points was by taking \(p\) and \(q\), and multiplying both by some factor r that i personally know. note also that thanks to the magic of elliptic curve pairings, checking that \(r = k \cdot s\) doesn't actually require knowing \(k\) instead, you can simply check whether or not \(e(r, q) = e(p, s)\). let's do something more interesting. suppose that we have ten pairs of points fall from the sky: \((p_1, q_1), (p_2, q_2)... (p_{10}, q_{10})\). in all cases, \(p_i \cdot k = q_i\). suppose that i then provide you with a pair of points \((r, s)\) where \(r \cdot k = s\). what do you know now? you know that \(r\) is some linear combination \(p_1 \cdot i_1 + p_2 \cdot i_2 + ... + p_{10} \cdot i_{10}\), where i know the coefficients \(i_1, i_2 ... i_{10}\). that is, the only way to arrive at such a pair of points \((r, s)\) is to take some multiples of \(p_1, p_2 ... p_{10}\) and add them together, and make the same calculation with \(q_1, q_2 ... q_{10}\). note that, given any specific set of \(p_1...p_{10}\) points that you might want to check linear combinations for, you can't actually create the accompanying \(q_1...q_{10}\) points without knowing what \(k\) is, and if you do know what \(k\) is then you can create a pair \((r, s)\) where \(r \cdot k = s\) for whatever \(r\) you want, without bothering to create a linear combination. hence, for this to work it's absolutely imperative that whoever creates those points is trustworthy and actually deletes \(k\) once they created the ten points. this is where the concept of a "trusted setup" comes from. remember that the solution to a qap is a set of polynomials \((a, b, c)\) such that \(a(x) \cdot b(x) c(x) = h(x) \cdot z(x)\), where: \(a\) is a linear combination of a set of polynomials \(\{a_1...a_m\}\) \(b\) is the linear combination of \(\{b_1...b_m\}\) with the same coefficients \(c\) is a linear combination of \(\{c_1...c_m\}\) with the same coefficients the sets \(\{a_1...a_m\}, \{b_1...b_m\}\) and \(\{c_1...c_m\}\) and the polynomial \(z\) are part of the problem statement. however, in most real-world cases, \(a, b\) and \(c\) are extremely large; for something with many thousands of circuit gates like a hash function, the polynomials (and the factors for the linear combinations) may have many thousands of terms. hence, instead of having the prover provide the linear combinations directly, we are going to use the trick that we introduced above to have the prover prove that they are providing something which is a linear combination, but without revealing anything else. you might have noticed that the trick above works on elliptic curve points, not polynomials. hence, what actually happens is that we add the following values to the trusted setup: \(g \cdot a_1(t), g \cdot a_1(t) \cdot k_a\) \(g \cdot a_2(t), g \cdot a_2(t) \cdot k_a\) ... \(g \cdot b_1(t), g \cdot b_1(t) \cdot k_b\) \(g \cdot b_2(t), g \cdot b_2(t) \cdot k_b\) ... \(g \cdot c_1(t), g \cdot c_1(t) \cdot k_c\) \(g \cdot c_2(t), g \cdot c_2(t) \cdot k_c\) ... you can think of t as a "secret point" at which the polynomial is evaluated. \(g\) is a "generator" (some random elliptic curve point that is specified as part of the protocol) and \(t, k_a, k_b\) and \(k_c\) are "toxic waste", numbers that absolutely must be deleted at all costs, or else whoever has them will be able to make fake proofs. now, if someone gives you a pair of points \(p\), \(q\) such that \(p \cdot k_a = q\) (reminder: we don't need \(k_a\) to check this, as we can do a pairing check), then you know that what they are giving you is a linear combination of \(a_i\) polynomials evaluated at \(t\). hence, so far the prover must give: \(\pi _a = g \cdot a(t), \pi '_a = g \cdot a(t) \cdot k_a\) \(\pi _b = g \cdot b(t), \pi '_b = g \cdot b(t) \cdot k_b\) \(\pi _c = g \cdot c(t), \pi '_c = g \cdot c(t) \cdot k_c\) note that the prover doesn't actually need to know (and shouldn't know!) \(t, k_a, k_b\) or \(k_c\) to compute these values; rather, the prover should be able to compute these values just from the points that we're adding to the trusted setup. the next step is to make sure that all three linear combinations have the same coefficients. this we can do by adding another set of values to the trusted setup: \(g \cdot (a_i(t) + b_i(t) + c_i(t)) \cdot b\), where \(b\) is another number that should be considered "toxic waste" and discarded as soon as the trusted setup is completed. we can then have the prover create a linear combination with these values with the same coefficients, and use the same pairing trick as above to verify that this value matches up with the provided \(a + b + c\). finally, we need to prove that \(a \cdot b c = h \cdot z\). we do this once again with a pairing check: \(e(\pi _a, \pi _b) / e(\pi _c, g) ?= e(\pi _h, g \cdot z(t))\) where \(\pi _h= g \cdot h(t)\). if the connection between this equation and \(a \cdot b c = h \cdot z\) does not make sense to you, go back and read the article on pairings. we saw above how to convert \(a, b\) and \(c\) into elliptic curve points; \(g\) is just the generator (ie. the elliptic curve point equivalent of the number one). we can add \(g \cdot z(t)\) to the trusted setup. \(h\) is harder; \(h\) is just a polynomial, and we predict very little ahead of time about what its coefficients will be for each individual qap solution. hence, we need to add yet more data to the trusted setup; specifically the sequence: \(g, g \cdot t, g \cdot t^2, g \cdot t^3, g \cdot t^4 ...\). in the zcash trusted setup, the sequence here goes up to about 2 million; this is how many powers of \(t\) you need to make sure that you will always be able to compute \(h(t)\), at least for the specific qap instance that they care about. and with that, the prover can provide all of the information for the verifier to make the final check. there is one more detail that we need to discuss. most of the time we don't just want to prove in the abstract that some solution exists for some specific problem; rather, we want to prove either the correctness of some specific solution (eg. proving that if you take the word "cow" and sha3 hash it a million times, the final result starts with 0x73064fe5), or that a solution exists if you restrict some of the parameters. for example, in a cryptocurrency instantiation where transaction amounts and account balances are encrypted, you want to prove that you know some decryption key k such that: decrypt(old_balance, k) >= decrypt(tx_value, k) decrypt(old_balance, k) decrypt(tx_value, k) = decrypt(new_balance, k) the encrypted old_balance, tx_value and new_balance should be specified publicly, as those are the specific values that we are looking to verify at that particular time; only the decryption key should be hidden. some slight modifications to the protocol are needed to create a "custom verification key" that corresponds to some specific restriction on the inputs. now, let's step back a bit. first of all, here's the verification algorithm in its entirety, courtesy of ben sasson, tromer, virza and chiesa: the first line deals with parametrization; essentially, you can think of its function as being to create a "custom verification key" for the specific instance of the problem where some of the arguments are specified. the second line is the linear combination check for \(a, b\) and \(c\); the third line is the check that the linear combinations have the same coefficients, and the fourth line is the product check \(a \cdot b c = h \cdot z\). altogether, the verification process is a few elliptic curve multiplications (one for each "public" input variable), and five pairing checks, one of which includes an additional pairing multiplication. the proof contains eight elliptic curve points: a pair of points each for \(a(t), b(t)\) and \(c(t)\), a point \(\pi _k\) for \(b \cdot (a(t) + b(t) + c(t))\), and a point \(\pi _h\) for \(h(t)\). seven of these points are on the \(f_p\) curve (32 bytes each, as you can compress the \(y\) coordinate to a single bit), and in the zcash implementation one point (\(\pi _b\)) is on the twisted curve in \(f_{p^2}\) (64 bytes), so the total size of the proof is ~288 bytes. the two computationally hardest parts of creating a proof are: dividing \((a \cdot b c) / z\) to get \(h\) (algorithms based on the fast fourier transform can do this in sub-quadratic time, but it's still quite computationally intensive) making the elliptic curve multiplications and additions to create the \(a(t), b(t), c(t)\) and \(h(t)\) values and their corresponding pairs the basic reason why creating a proof is so hard is the fact that what was a single binary logic gate in the original computation turns into an operation that must be cryptographically processed through elliptic curve operations if we are making a zero-knowledge proof out of it. this fact, together with the superlinearity of fast fourier transforms, means that proof creation takes ~20–40 seconds for a zcash transaction. another very important question is: can we try to make the trusted setup a little... less trust-demanding? unfortunately we can't make it completely trustless; the koe assumption itself precludes making independent pairs \((p_i, p_i \cdot k)\) without knowing what \(k\) is. however, we can increase security greatly by using \(n\)-of-\(n\) multiparty computation that is, constructing the trusted setup between \(n\) parties in such a way that as long as at least one of the participants deleted their toxic waste then you're okay. to get a bit of a feel for how you would do this, here's a simple algorithm for taking an existing set (\(g, g \cdot t, g \cdot t^2, g \cdot t^3...\)), and "adding in" your own secret so that you need both your secret and the previous secret (or previous set of secrets) to cheat. the output set is simply: \(g, (g \cdot t) \cdot s, (g \cdot t^2) \cdot s^2, (g \cdot t^3) \cdot s^3...\) note that you can produce this set knowing only the original set and s, and the new set functions in the same way as the old set, except now using \(t \cdot s\) as the "toxic waste" instead of \(t\). as long as you and the person (or people) who created the previous set do not both fail to delete your toxic waste and later collude, the set is "safe". doing this for the complete trusted setup is quite a bit harder, as there are several values involved, and the algorithm has to be done between the parties in several rounds. it's an area of active research to see if the multi-party computation algorithm can be simplified further and made to require fewer rounds or made more parallelizable, as the more you can do that the more parties it becomes feasible to include into the trusted setup procedure. it's reasonable to see why a trusted setup between six participants who all know and work with each other might make some people uncomfortable, but a trusted setup with thousands of participants would be nearly indistinguishable from no trust at all and if you're really paranoid, you can get in and participate in the setup procedure yourself, and be sure that you personally deleted your value. another area of active research is the use of other approaches that do not use pairings and the same trusted setup paradigm to achieve the same goal; see eli ben sasson's recent presentation for one alternative (though be warned, it's at least as mathematically complicated as snarks are!) special thanks to ariel gabizon and christian reitwiessner for reviewing. dark mode toggle proof of stake faq 2017 dec 31 see all posts contents what is proof of stake what are the benefits of proof of stake as opposed to proof of work? how does proof of stake fit into traditional byzantine fault tolerance research? what is the "nothing at stake" problem and how can it be fixed? that shows how chain-based algorithms solve nothing-at-stake. now how do bft-style proof of stake algorithms work? what is "economic finality" in general? so how does this relate to byzantine fault tolerance theory? what is "weak subjectivity"? can we try to automate the social authentication to reduce the load on users? can one economically penalize censorship in proof of stake? how does validator selection work, and what is stake grinding? what would the equivalent of a 51% attack against casper look like? that sounds like a lot of reliance on out-of-band social coordination; is that not dangerous? doesn't mc <= mr mean that all consensus algorithms with a given security level are equally efficient (or in other words, equally wasteful)? what about capital lockup costs? will exchanges in proof of stake pose a similar centralization risk to pools in proof of work? are there economic ways to discourage centralization? can proof of stake be used in private/consortium chains? can multi-currency proof of stake work? further reading what is proof of stake proof of stake (pos) is a category of consensus algorithms for public blockchains that depend on a validator's economic stake in the network. in proof of work (pow) based public blockchains (e.g. bitcoin and the current implementation of ethereum), the algorithm rewards participants who solve cryptographic puzzles in order to validate transactions and create new blocks (i.e. mining). in pos-based public blockchains (e.g. ethereum's upcoming casper implementation), a set of validators take turns proposing and voting on the next block, and the weight of each validator's vote depends on the size of its deposit (i.e. stake). significant advantages of pos include security, reduced risk of centralization, and energy efficiency. in general, a proof of stake algorithm looks as follows. the blockchain keeps track of a set of validators, and anyone who holds the blockchain's base cryptocurrency (in ethereum's case, ether) can become a validator by sending a special type of transaction that locks up their ether into a deposit. the process of creating and agreeing to new blocks is then done through a consensus algorithm that all current validators can participate in. there are many kinds of consensus algorithms, and many ways to assign rewards to validators who participate in the consensus algorithm, so there are many "flavors" of proof of stake. from an algorithmic perspective, there are two major types: chain-based proof of stake and bft-style proof of stake. in chain-based proof of stake, the algorithm pseudo-randomly selects a validator during each time slot (e.g. every period of 10 seconds might be a time slot), and assigns that validator the right to create a single block, and this block must point to some previous block (normally the block at the end of the previously longest chain), and so over time most blocks converge into a single constantly growing chain. in bft-style proof of stake, validators are randomly assigned the right to propose blocks, but agreeing on which block is canonical is done through a multi-round process where every validator sends a "vote" for some specific block during each round, and at the end of the process all (honest and online) validators permanently agree on whether or not any given block is part of the chain. note that blocks may still be chained together; the key difference is that consensus on a block can come within one block, and does not depend on the length or size of the chain after it. what are the benefits of proof of stake as opposed to proof of work? see a proof of stake design philosophy for a more long-form argument. in short: no need to consume large quantities of electricity in order to secure a blockchain (e.g. it's estimated that both bitcoin and ethereum burn over $1 million worth of electricity and hardware costs per day as part of their consensus mechanism). because of the lack of high electricity consumption, there is not as much need to issue as many new coins in order to motivate participants to keep participating in the network. it may theoretically even be possible to have negative net issuance, where a portion of transaction fees is "burned" and so the supply goes down over time. proof of stake opens the door to a wider array of techniques that use game-theoretic mechanism design in order to better discourage centralized cartels from forming and, if they do form, from acting in ways that are harmful to the network (e.g. like selfish mining in proof of work). reduced centralization risks, as economies of scale are much less of an issue. $10 million of coins will get you exactly 10 times higher returns than $1 million of coins, without any additional disproportionate gains because at the higher level you can afford better mass-production equipment, which is an advantage for proof-of-work. ability to use economic penalties to make various forms of 51% attacks vastly more expensive to carry out than proof of work to paraphrase vlad zamfir, "it's as though your asic farm burned down if you participated in a 51% attack". how does proof of stake fit into traditional byzantine fault tolerance research? there are several fundamental results from byzantine fault tolerance research that apply to all consensus algorithms, including traditional consensus algorithms like pbft but also any proof of stake algorithm and, with the appropriate mathematical modeling, proof of work. the key results include: cap theorem "in the cases that a network partition takes place, you have to choose either consistency or availability, you cannot have both". the intuitive argument is simple: if the network splits in half, and in one half i send a transaction "send my 10 coins to a" and in the other i send a transaction "send my 10 coins to b", then either the system is unavailable, as one or both transactions will not be processed, or it becomes inconsistent, as one half of the network will see the first transaction completed and the other half will see the second transaction completed. note that the cap theorem has nothing to do with scalability; it applies to sharded and non-sharded systems equally. see also https://github.com/ethereum/wiki/wiki/sharding-faqs#but-doesnt-the-cap-theorem-mean-that-fully-secure-distributed-systems-are-impossible-and-so-sharding-is-futile. flp impossibility in an asynchronous setting (i.e. there are no guaranteed bounds on network latency even between correctly functioning nodes), it is not possible to create an algorithm which is guaranteed to reach consensus in any specific finite amount of time if even a single faulty/dishonest node is present. note that this does not rule out "las vegas" algorithms that have some probability each round of achieving consensus and thus will achieve consensus within t seconds with probability exponentially approaching 1 as t grows; this is in fact the "escape hatch" that many successful consensus algorithms use. bounds on fault tolerance from the dls paper we have: (i) protocols running in a partially synchronous network model (i.e. there is a bound on network latency but we do not know ahead of time what it is) can tolerate up to 1/3 arbitrary (i.e. "byzantine") faults, (ii) deterministic protocols in an asynchronous model (i.e. no bounds on network latency) cannot tolerate faults (although their paper fails to mention that randomized algorithms can with up to 1/3 fault tolerance), (iii) protocols in a synchronous model (i.e. network latency is guaranteed to be less than a known d) can, surprisingly, tolerate up to 100% fault tolerance, although there are restrictions on what can happen when more than or equal to 1/2 of nodes are faulty. note that the "authenticated byzantine" model is the one worth considering, not the "byzantine" one; the "authenticated" part essentially means that we can use public key cryptography in our algorithms, which is in modern times very well-researched and very cheap. proof of work has been rigorously analyzed by andrew miller and others and fits into the picture as an algorithm reliant on a synchronous network model. we can model the network as being made up of a near-infinite number of nodes, with each node representing a very small unit of computing power and having a very small probability of being able to create a block in a given period. in this model, the protocol has 50% fault tolerance assuming zero network latency, ~46% (ethereum) and ~49.5% (bitcoin) fault tolerance under actually observed conditions, but goes down to 33% if network latency is equal to the block time, and reduces to zero as network latency approaches infinity. proof of stake consensus fits more directly into the byzantine fault tolerant consensus mould, as all validators have known identities (stable ethereum addresses) and the network keeps track of the total size of the validator set. there are two general lines of proof of stake research, one looking at synchronous network models and one looking at partially asynchronous network models. "chain-based" proof of stake algorithms almost always rely on synchronous network models, and their security can be formally proven within these models similarly to how security of proof of work algorithms can be proven. a line of research connecting traditional byzantine fault tolerant consensus in partially synchronous networks to proof of stake also exists, but is more complex to explain; it will be covered in more detail in later sections. proof of work algorithms and chain-based proof of stake algorithms choose availability over consistency, but bft-style consensus algorithms lean more toward consistency; tendermint chooses consistency explicitly, and casper uses a hybrid model that prefers availability but provides as much consistency as possible and makes both on-chain applications and clients aware of how strong the consistency guarantee is at any given time. note that ittay eyal and emin gun sirer's selfish mining discovery, which places 25% and 33% bounds on the incentive compatibility of bitcoin mining depending on the network model (i.e. mining is only incentive compatible if collusions larger than 25% or 33% are impossible) has nothing to do with results from traditional consensus algorithm research, which does not touch incentive compatibility. what is the "nothing at stake" problem and how can it be fixed? in many early (all chain-based) proof of stake algorithms, including peercoin, there are only rewards for producing blocks, and no penalties. this has the unfortunate consequence that, in the case that there are multiple competing chains, it is in a validator's incentive to try to make blocks on top of every chain at once, just to be sure: in proof of work, doing so would require splitting one's computing power in half, and so would not be lucrative: the result is that if all actors are narrowly economically rational, then even if there are no attackers, a blockchain may never reach consensus. if there is an attacker, then the attacker need only overpower altruistic nodes (who would exclusively stake on the original chain), and not rational nodes (who would stake on both the original chain and the attacker's chain), in contrast to proof of work, where the attacker must overpower both altruists and rational nodes (or at least credibly threaten to: see p + epsilon attacks). some argue that stakeholders have an incentive to act correctly and only stake on the longest chain in order to "preserve the value of their investment", however this ignores that this incentive suffers from tragedy of the commons problems: each individual stakeholder might only have a 1% chance of being "pivotal" (i.e. being in a situation where if they participate in an attack then it succeeds and if they do not participate it fails), and so the bribe needed to convince them personally to join an attack would be only 1% of the size of their deposit; hence, the required combined bribe would be only 0.5-1% of the total sum of all deposits. additionally, this argument implies that any zero-chance-of-failure situation is not a stable equilibrium, as if the chance of failure is zero then everyone has a 0% chance of being pivotal. this can be solved via two strategies. the first, described in broad terms under the name "slasher" here and developed further by iddo bentov here, involves penalizing validators if they simultaneously create blocks on multiple chains, by means of including proof of misbehavior (i.e. two conflicting signed block headers) into the blockchain as a later point in time at which point the malfeasant validator's deposit is deducted appropriately. this changes the incentive structure thus: note that for this algorithm to work, the validator set needs to be determined well ahead of time. otherwise, if a validator has 1% of the stake, then if there are two branches a and b then 0.99% of the time the validator will be eligible to stake only on a and not on b, 0.99% of the time the validator will be eligible to stake on b and not on a, and only 0.01% of the time will the validator will be eligible to stake on both. hence, the validator can with 99% efficiency probabilistically double-stake: stake on a if possible, stake on b if possible, and only if the choice between both is open stake on the longer chain. this can only be avoided if the validator selection is the same for every block on both branches, which requires the validators to be selected at a time before the fork takes place. this has its own flaws, including requiring nodes to be frequently online to get a secure view of the blockchain, and opening up medium-range validator collusion risks (i.e. situations where, for example, 25 out of 30 consecutive validators get together and agree ahead of time to implement a 51% attack on the previous 19 blocks), but if these risks are deemed acceptable then it works well. the second strategy is to simply punish validators for creating blocks on the wrong chain. that is, if there are two competing chains, a and b, then if a validator creates a block on b, they get a reward of +r on b, but the block header can be included into a (in casper this is called a "dunkle") and on a the validator suffers a penalty of -f (possibly f = r). this changes the economic calculation thus: the intuition here is that we can replicate the economics of proof of work inside of proof of stake. in proof of work, there is also a penalty for creating a block on the wrong chain, but this penalty is implicit in the external environment: miners have to spend extra electricity and obtain or rent extra hardware. here, we simply make the penalties explicit. this mechanism has the disadvantage that it imposes slightly more risk on validators (although the effect should be smoothed out over time), but has the advantage that it does not require validators to be known ahead of time. that shows how chain-based algorithms solve nothing-at-stake. now how do bft-style proof of stake algorithms work? bft-style (partially synchronous) proof of stake algorithms allow validators to "vote" on blocks by sending one or more types of signed messages, and specify two kinds of rules: finality conditions rules that determine when a given hash can be considered finalized. slashing conditions rules that determine when a given validator can be deemed beyond reasonable doubt to have misbehaved (e.g. voting for multiple conflicting blocks at the same time). if a validator triggers one of these rules, their entire deposit gets deleted. to illustrate the different forms that slashing conditions can take, we will give two examples of slashing conditions (hereinafter, "2/3 of all validators" is shorthand for "2/3 of all validators weighted by deposited coins", and likewise for other fractions and percentages). in these examples, "prepare" and "commit" should be understood as simply referring to two types of messages that validators can send. if messages contains messages of the form ["commit", hash1, view] and ["commit", hash2, view] for the same view but differing hash1 and hash2 signed by the same validator, then that validator is slashed. if messages contains a message of the form ["commit", hash, view1], then unless either view1 = -1 or there also exist messages of the form ["prepare", hash, view1, view2] for some specific view2, where view2 < view1, signed by 2/3 of all validators, then the validator that made the commit is slashed. there are two important desiderata for a suitable set of slashing conditions to have: accountable safety if conflicting hash1 and hash2 (i.e. hash1 and hash2 are different, and neither is a descendant of the other) are finalized, then at least 1/3 of all validators must have violated some slashing condition. plausible liveness unless at least 1/3 of all validators have violated some slashing condition, there exists a set of messages that 2/3 of validators can produce that finalize some value. if we have a set of slashing conditions that satisfies both properties, then we can incentivize participants to send messages, and start benefiting from economic finality. what is "economic finality" in general? economic finality is the idea that once a block is finalized, or more generally once enough messages of certain types have been signed, then the only way that at any point in the future the canonical history will contain a conflicting block is if a large number of people are willing to burn very large amounts of money. if a node sees that this condition has been met for a given block, then they have a very economically strong assurance that that block will always be part of the canonical history that everyone agrees on. there are two "flavors" of economic finality: a block can be economically finalized if a sufficient number of validators have signed cryptoeconomic claims of the form "i agree to lose x in all histories where block b is not included". this gives clients assurance that either (i) b is part of the canonical chain, or (ii) validators lost a large amount of money in order to trick them into thinking that this is the case. a block can be economically finalized if a sufficient number of validators have signed messages expressing support for block b, and there is a mathematical proof that if some b' != b is also finalized under the same definition then validators lose a large amount of money. if clients see this, and also validate the chain, and validity plus finality is a sufficient condition for precedence in the canonical fork choice rule, then they get an assurance that either (i) b is part of the canonical chain, or (ii) validators lost a large amount of money in making a conflicting chain that was also finalized. the two approaches to finality inherit from the two solutions to the nothing at stake problem: finality by penalizing incorrectness, and finality by penalizing equivocation. the main benefit of the first approach is that it is more light-client friendly and is simpler to reason about, and the main benefits of the second approach are that (i) it's easier to see that honest validators will not be punished, and (ii) griefing factors are more favorable to honest validators. casper follows the second flavor, though it is possible that an on-chain mechanism will be added where validators can voluntarily opt-in to signing finality messages of the first flavor, thereby enabling much more efficient light clients. so how does this relate to byzantine fault tolerance theory? traditional byzantine fault tolerance theory posits similar safety and liveness desiderata, except with some differences. first of all, traditional byzantine fault tolerance theory simply requires that safety is achieved if 2/3 of validators are honest. this is a strictly easier model to work in; traditional fault tolerance tries to prove "if mechanism m has a safety failure, then at least 1/3 of nodes are faulty", whereas our model tries to prove "if mechanism m has a safety failure, then at least 1/3 of nodes are faulty, and you know which ones, even if you were offline at the time the failure took place". from a liveness perspective, our model is the easier one, as we do not demand a proof that the network will come to consensus, we just demand a proof that it does not get stuck. fortunately, we can show the additional accountability requirement is not a particularly difficult one; in fact, with the right "protocol armor", we can convert any traditional partially synchronous or asynchronous byzantine fault-tolerant algorithm into an accountable algorithm. the proof of this basically boils down to the fact that faults can be exhaustively categorized into a few classes, and each one of these classes is either accountable (i.e. if you commit that type of fault you can get caught, so we can make a slashing condition for it) or indistinguishable from latency (note that even the fault of sending messages too early is indistinguishable from latency, as one can model it by speeding up everyone's clocks and assigning the messages that weren't sent too early a higher latency). what is "weak subjectivity"? it is important to note that the mechanism of using deposits to ensure there is "something at stake" does lead to one change in the security model. suppose that deposits are locked for four months, and can later be withdrawn. suppose that an attempted 51% attack happens that reverts 10 days worth of transactions. the blocks created by the attackers can simply be imported into the main chain as proof-of-malfeasance (or "dunkles") and the validators can be punished. however, suppose that such an attack happens after six months. then, even though the blocks can certainly be re-imported, by that time the malfeasant validators will be able to withdraw their deposits on the main chain, and so they cannot be punished. to solve this problem, we introduce a "revert limit" a rule that nodes must simply refuse to revert further back in time than the deposit length (i.e. in our example, four months), and we additionally require nodes to log on at least once every deposit length to have a secure view of the chain. note that this rule is different from every other consensus rule in the protocol, in that it means that nodes may come to different conclusions depending on when they saw certain messages. the time that a node saw a given message may be different between different nodes; hence we consider this rule "subjective" (alternatively, one well-versed in byzantine fault tolerance theory may view it as a kind of synchrony assumption). however, the "subjectivity" here is very weak: in order for a node to get on the "wrong" chain, they must receive the original message four months later than they otherwise would have. this is only possible in two cases: when a node connects to the blockchain for the first time. if a node has been offline for more than four months. we can solve (1) by making it the user's responsibility to authenticate the latest state out of band. they can do this by asking their friends, block explorers, businesses that they interact with, etc. for a recent block hash in the chain that they see as the canonical one. in practice, such a block hash may well simply come as part of the software they use to verify the blockchain; an attacker that can corrupt the checkpoint in the software can arguably just as easily corrupt the software itself, and no amount of pure cryptoeconomic verification can solve that problem. (2) does genuinely add an additional security requirement for nodes, though note once again that the possibility of hard forks and security vulnerabilities, and the requirement to stay up to date to know about them and install any needed software updates, exists in proof of work too. note that all of this is a problem only in the very limited case where a majority of previous stakeholders from some point in time collude to attack the network and create an alternate chain; most of the time we expect there will only be one canonical chain to choose from. can we try to automate the social authentication to reduce the load on users? one approach is to bake it into natural user workflow: a bip 70-style payment request could include a recent block hash, and the user's client software would make sure that they are on the same chain as the vendor before approving a payment (or for that matter, any on-chain interaction). the other is to use jeff coleman's universal hash time. if uht is used, then a successful attack chain would need to be generated secretly at the same time as the legitimate chain was being built, requiring a majority of validators to secretly collude for that long. can one economically penalize censorship in proof of stake? unlike reverts, censorship is much more difficult to prove. the blockchain itself cannot directly tell the difference between "user a tried to send transaction x but it was unfairly censored", "user a tried to send transaction x but it never got in because the transaction fee was insufficient" and "user a never tried to send transaction x at all". see also a note on data availability and erasure codes. however, there are a number of techniques that can be used to mitigate censorship issues. the first is censorship resistance by halting problem. in the weaker version of this scheme, the protocol is designed to be turing-complete in such a way that a validator cannot even tell whether or not a given transaction will lead to an undesired action without spending a large amount of processing power executing the transaction, and thus opening itself up to denial-of-service attacks. this is what prevented the dao soft fork. in the stronger version of the scheme, transactions can trigger guaranteed effects at some point in the near to mid-term future. hence, a user could send multiple transactions which interact with each other and with predicted third-party information to lead to some future event, but the validators cannot possibly tell that this is going to happen until the transactions are already included (and economically finalized) and it is far too late to stop them; even if all future transactions are excluded, the event that validators wish to halt would still take place. note that in this scheme, validators could still try to prevent all transactions, or perhaps all transactions that do not come packaged with some formal proof that they do not lead to anything undesired, but this would entail forbidding a very wide class of transactions to the point of essentially breaking the entire system, which would cause validators to lose value as the price of the cryptocurrency in which their deposits are denominated would drop. the second, described by adam back here, is to require transactions to be timelock-encrypted. hence, validators will include the transactions without knowing the contents, and only later could the contents automatically be revealed, by which point once again it would be far too late to un-include the transactions. if validators were sufficiently malicious, however, they could simply only agree to include transactions that come with a cryptographic proof (e.g. zk-snark) of what the decrypted version is; this would force users to download new client software, but an adversary could conveniently provide such client software for easy download, and in a game-theoretic model users would have the incentive to play along. perhaps the best that can be said in a proof-of-stake context is that users could also install a software update that includes a hard fork that deletes the malicious validators and this is not that much harder than installing a software update to make their transactions "censorship-friendly". hence, all in all this scheme is also moderately effective, though it does come at the cost of slowing interaction with the blockchain down (note that the scheme must be mandatory to be effective; otherwise malicious validators could much more easily simply filter encrypted transactions without filtering the quicker unencrypted transactions). a third alternative is to include censorship detection in the fork choice rule. the idea is simple. nodes watch the network for transactions, and if they see a transaction that has a sufficiently high fee for a sufficient amount of time, then they assign a lower "score" to blockchains that do not include this transaction. if all nodes follow this strategy, then eventually a minority chain would automatically coalesce that includes the transactions, and all honest online nodes would follow it. the main weakness of such a scheme is that offline nodes would still follow the majority branch, and if the censorship is temporary and they log back on after the censorship ends then they would end up on a different branch from online nodes. hence, this scheme should be viewed more as a tool to facilitate automated emergency coordination on a hard fork than something that would play an active role in day-to-day fork choice. how does validator selection work, and what is stake grinding? in any chain-based proof of stake algorithm, there is a need for some mechanism which randomly selects which validator out of the currently active validator set can make the next block. for example, if the currently active validator set consists of alice with 40 ether, bob with 30 ether, charlie with 20 ether and david with 10 ether, then you want there to be a 40% chance that alice will be the next block creator, 30% chance that bob will be, etc (in practice, you want to randomly select not just one validator, but rather an infinite sequence of validators, so that if alice doesn't show up there is someone who can replace her after some time, but this doesn't change the fundamental problem). in non-chain-based algorithms randomness is also often needed for different reasons. "stake grinding" is a class of attack where a validator performs some computation or takes some other step to try to bias the randomness in their own favor. for example: in peercoin, a validator could "grind" through many combinations of parameters and find favorable parameters that would increase the probability of their coins generating a valid block. in one now-defunct implementation, the randomness for block n+1 was dependent on the signature of block n. this allowed a validator to repeatedly produce new signatures until they found one that allowed them to get the next block, thereby seizing control of the system forever. in nxt, the randomness for block n+1 is dependent on the validator that creates block n. this allows a validator to manipulate the randomness by simply skipping an opportunity to create a block. this carries an opportunity cost equal to the block reward, but sometimes the new random seed would give the validator an above-average number of blocks over the next few dozen blocks. see here and here for a more detailed analysis. and (2) are easy to solve; the general approach is to require validators to deposit their coins well in advance, and not to use information that can be easily manipulated as source data for the randomness. there are several main strategies for solving problems like (3). the first is to use schemes based on secret sharing or deterministic threshold signatures and have validators collaboratively generate the random value. these schemes are robust against all manipulation unless a majority of validators collude (in some cases though, depending on the implementation, between 33-50% of validators can interfere in the operation, leading to the protocol having a 67% liveness assumption). the second is to use cryptoeconomic schemes where validators commit to information (i.e. publish sha3(x)) well in advance, and then must publish x in the block; x is then added into the randomness pool. there are two theoretical attack vectors against this: manipulate x at commitment time. this is impractical because the randomness result would take many actors' values into account, and if even one of them is honest then the output will be a uniform distribution. a uniform distribution xored together with arbitrarily many arbitrarily biased distributions still gives a uniform distribution. selectively avoid publishing blocks. however, this attack costs one block reward of opportunity cost, and because the scheme prevents anyone from seeing any future validators except for the next, it almost never provides more than one block reward worth of revenue. the only exception is the case where, if a validator skips, the next validator in line and the first child of that validator will both be the same validator; if these situations are a grave concern then we can punish skipping further via an explicit skipping penalty. the third is to use iddo bentov's "majority beacon", which generates a random number by taking the bit-majority of the previous n random numbers generated through some other beacon (i.e. the first bit of the result is 1 if the majority of the first bits in the source numbers is 1 and otherwise it's 0, the second bit of the result is 1 if the majority of the second bits in the source numbers is 1 and otherwise it's 0, etc). this gives a cost-of-exploitation of ~c * sqrt(n) where c is the cost of exploitation of the underlying beacons. hence, all in all, many known solutions to stake grinding exist; the problem is more like differential cryptanalysis than the halting problem an annoyance that proof of stake designers eventually understood and now know how to overcome, not a fundamental and inescapable flaw. what would the equivalent of a 51% attack against casper look like? there are four basic types of 51% attack: finality reversion: validators that already finalized block a then finalize some competing block a', thereby breaking the blockchain's finality guarantee. invalid chain finalization: validators finalize an invalid (or unavailable) block. liveness denial: validators stop finalizing blocks. censorship: validators block some or all transactions or blocks from entering the chain. in the first case, users can socially coordinate out-of-band to agree which finalized block came first, and favor that block. the second case can be solved with fraud proofs and data availability proofs. the third case can be solved by a modification to proof of stake algorithms that gradually reduces ("leaks") non-participating nodes' weights in the validator set if they do not participate in consensus; the casper ffg paper includes a description of this. the fourth is most difficult. the fourth can be recovered from via a "minority soft fork", where a minority of honest validators agree the majority is censoring them, and stop building on their chain. instead, they continue their own chain, and eventually the "leak" mechanism described above ensures that this honest minority becomes a 2/3 supermajority on the new chain. at that point, the market is expected to favor the chain controlled by honest nodes over the chain controlled by dishonest nodes. that sounds like a lot of reliance on out-of-band social coordination; is that not dangerous? attacks against casper are extremely expensive; as we will see below, attacks against casper cost as much, if not more, than the cost of buying enough mining power in a proof of work chain to permanently 51% attack it over and over again to the point of uselessness. hence, the recovery techniques described above will only be used in very extreme circumstances; in fact, advocates of proof of work also generally express willingness to use social coordination in similar circumstances by, for example, changing the proof of work algorithm. hence, it is not even clear that the need for social coordination in proof of stake is larger than it is in proof of work. in reality, we expect the amount of social coordination required to be near-zero, as attackers will realize that it is not in their benefit to burn such large amounts of money to simply take a blockchain offline for one or two days. doesn't mc <= mr mean that all consensus algorithms with a given security level are equally efficient (or in other words, equally wasteful)? this is an argument that many have raised, perhaps best explained by paul sztorc in this article. essentially, if you create a way for people to earn $100, then people will be willing to spend anywhere up to $99.9 (including the cost of their own labor) in order to get it; marginal cost approaches marginal revenue. hence, the theory goes, any algorithm with a given block reward will be equally "wasteful" in terms of the quantity of socially unproductive activity that is carried out in order to try to get the reward. there are three flaws with this: it's not enough to simply say that marginal cost approaches marginal revenue; one must also posit a plausible mechanism by which someone can actually expend that cost. for example, if tomorrow i announce that every day from then on i will give $100 to a randomly selected one of a given list of ten people (using my laptop's /dev/urandom as randomness), then there is simply no way for anyone to send $99 to try to get at that randomness. either they are not in the list of ten, in which case they have no chance no matter what they do, or they are in the list of ten, in which case they don't have any reasonable way to manipulate my randomness so they're stuck with getting the expected-value $10 per day. mc <= mr does not imply total cost approaches total revenue. for example, suppose that there is an algorithm which pseudorandomly selects 1000 validators out of some very large set (each validator getting a reward of $1), you have 10% of the stake so on average you get 100, and at a cost of $1 you can force the randomness to reset (and you can repeat this an unlimited number of times). due to the central limit theorem, the standard deviation of your reward is $10, and based on other known results in math the expected maximum of n random samples is slightly under m + s * sqrt(2 * log(n)) where m is the mean and s is the standard deviation. hence the reward for making additional trials (i.e. increasing n) drops off sharply, e.g. with 0 re-trials your expected reward is $100, with one re-trial it's $105.5, with two it's $108.5, with three it's $110.3, with four it's $111.6, with five it's $112.6 and with six it's $113.5. hence, after five retrials it stops being worth it. as a result, an economically motivated attacker with ten percent of stake will inefficiently spend $5 to get an additional revenue of $13, though the total revenue is $113. if the exploitable mechanisms only expose small opportunities, the economic loss will be small; it is decidedly not the case that a single drop of exploitability brings the entire flood of pow-level economic waste rushing back in. this point will also be very relevant in our below discussion on capital lockup costs. proof of stake can be secured with much lower total rewards than proof of work. what about capital lockup costs? locking up x ether in a deposit is not free; it entails a sacrifice of optionality for the ether holder. right now, if i have 1000 ether, i can do whatever i want with it; if i lock it up in a deposit, then it's stuck there for months, and i do not have, for example, the insurance utility of the money being there to pay for sudden unexpected expenses. i also lose some freedom to change my token allocations away from ether within that timeframe; i could simulate selling ether by shorting an amount equivalent to the deposit on an exchange, but this itself carries costs including exchange fees and paying interest. some might argue: isn't this capital lockup inefficiency really just a highly indirect way of achieving the exact same level of economic inefficiency as exists in proof of work? the answer is no, for both reasons (2) and (3) above. let us start with (3) first. consider a model where proof of stake deposits are infinite-term, asics last forever, asic technology is fixed (i.e. no moore's law) and electricity costs are zero. let's say the equilibrium interest rate is 5% per annum. in a proof of work blockchain, i can take $1000, convert it into a miner, and the miner will pay me $50 in rewards per year forever. in a proof of stake blockchain, i would buy $1000 of coins, deposit them (i.e. losing them forever), and get $50 in rewards per year forever. so far, the situation looks completely symmetrical (technically, even here, in the proof of stake case my destruction of coins isn't fully socially destructive as it makes others' coins worth more, but we can leave that aside for the moment). the cost of a "maginot-line" 51% attack (i.e. buying up more hardware than the rest of the network) increases by $1000 in both cases. now, let's perform the following changes to our model in turn: moore's law exists, asics depreciate by 50% every 2.772 years (that's a continuously-compounded 25% annual depreciation; picked to make the numbers simpler). if i want to retain the same "pay once, get money forever" behavior, i can do so: i would put $1000 into a fund, where $167 would go into an asic and the remaining $833 would go into investments at 5% interest; the $41.67 dividends per year would be just enough to keep renewing the asic hardware (assuming technological development is fully continuous, once again to make the math simpler). rewards would go down to $8.33 per year; hence, 83.3% of miners will drop out until the system comes back into equilibrium with me earning $50 per year, and so the maginot-line cost of an attack on pow given the same rewards drops by a factor of 6. electricity plus maintenance makes up 1/3 of mining costs. we estimate the 1/3 from recent mining statistics: one of bitfury's new data centers consumes 0.06 joules per gigahash, or 60 j/th or 0.000017 kwh/th, and if we assume the entire bitcoin network has similar efficiencies we get 27.9 kwh per second given 1.67 million th/s total bitcoin hashpower. electricity in china costs $0.11 per kwh, so that's about $3 per second, or $260,000 per day. bitcoin block rewards plus fees are $600 per btc * 13 btc per block * 144 blocks per day = $1.12m per day. thus electricity itself would make up 23% of costs, and we can back-of-the-envelope estimate maintenance at 10% to give a clean 1/3 ongoing costs, 2/3 fixed costs split. this means that out of your $1000 fund, only $111 would go into the asic, $56 would go into paying ongoing costs, and $833 would go into investments; hence the maginot-line cost of attack is 9x lower than in our original setting. deposits are temporary, not permanent. sure, if i voluntarily keep staking forever, then this changes nothing. however, i regain some of the optionality that i had before; i could quit within a medium timeframe (say, 4 months) at any time. this means that i would be willing to put more than $1000 of ether in for the $50 per year gain; perhaps in equilibrium it would be something like $3000. hence, the cost of the maginot line attack on pos increases by a factor of three, and so on net pos gives 27x more security than pow for the same cost. the above included a large amount of simplified modeling, however it serves to show how multiple factors stack up heavily in favor of pos in such a way that pos gets more bang for its buck in terms of security. the meta-argument for why this perhaps suspiciously multifactorial argument leans so heavily in favor of pos is simple: in pow, we are working directly with the laws of physics. in pos, we are able to design the protocol in such a way that it has the precise properties that we want in short, we can optimize the laws of physics in our favor. the "hidden trapdoor" that gives us (3) is the change in the security model, specifically the introduction of weak subjectivity. now, we can talk about the marginal/total distinction. in the case of capital lockup costs, this is very important. for example, consider a case where you have $100,000 of ether. you probably intend to hold a large portion of it for a long time; hence, locking up even $50,000 of the ether should be nearly free. locking up $80,000 would be slightly more inconvenient, but $20,000 of breathing room still gives you a large space to maneuver. locking up $90,000 is more problematic, $99,000 is very problematic, and locking up all $100,000 is absurd, as it means you would not even have a single bit of ether left to pay basic transaction fees. hence, your marginal costs increase quickly. we can show the difference between this state of affairs and the state of affairs in proof of work as follows: hence, the total cost of proof of stake is potentially much lower than the marginal cost of depositing 1 more eth into the system multiplied by the amount of ether currently deposited. note that this component of the argument unfortunately does not fully translate into reduction of the "safe level of issuance". it does help us because it shows that we can get substantial proof of stake participation even if we keep issuance very low; however, it also means that a large portion of the gains will simply be borne by validators as economic surplus. will exchanges in proof of stake pose a similar centralization risk to pools in proof of work? from a centralization perspective, in both bitcoin and ethereum it's the case that roughly three pools are needed to coordinate on a 51% attack (4 in bitcoin, 3 in ethereum at the time of this writing). in pos, if we assume 30% participation including all exchanges, then three exchanges would be enough to make a 51% attack; if participation goes up to 40% then the required number goes up to eight. however, exchanges will not be able to participate with all of their ether; the reason is that they need to accomodate withdrawals. additionally, pooling in pos is discouraged because it has a much higher trust requirement a proof of stake pool can pretend to be hacked, destroy its participants' deposits and claim a reward for it. on the other hand, the ability to earn interest on one's coins without oneself running a node, even if trust is required, is something that many may find attractive; all in all, the centralization balance is an empirical question for which the answer is unclear until the system is actually running for a substantial period of time. with sharding, we expect pooling incentives to reduce further, as (i) there is even less concern about variance, and (ii) in a sharded model, transaction verification load is proportional to the amount of capital that one puts in, and so there are no direct infrastructure savings from pooling. a final point is that centralization is less harmful in proof of stake than in proof of work, as there are much cheaper ways to recover from successful 51% attacks; one does not need to switch to a new mining algorithm. are there economic ways to discourage centralization? one strategy suggested by vlad zamfir is to only partially destroy deposits of validators that get slashed, setting the percentage destroyed to be proportional to the percentage of other validators that have been slashed recently. this ensures that validators lose all of their deposits in the event of an actual attack, but only a small part of their deposits in the event of a one-off mistake. this makes lower-security staking strategies possible, and also specifically incentivizes validators to have their errors be as uncorrelated (or ideally, anti-correlated) with other validators as possible; this involves not being in the largest pool, putting one's node on the largest virtual private server provider and even using secondary software implementations, all of which increase decentralization. can proof of stake be used in private/consortium chains? generally, yes; any proof of stake algorithm can be used as a consensus algorithm in private/consortium chain settings. the only change is that the way the validator set is selected would be different: it would start off as a set of trusted users that everyone agrees on, and then it would be up to the validator set to vote on adding in new validators. can multi-currency proof of stake work? there has been a lot of interest in proof of stake protocols where users can stake any currency, or one of multiple currencies. however, these designs unfortunately introduce economic challenges that likely make them much more trouble than any benefit that could be received from them. the key problems include: price oracle dependence: if people are staking in multiple cryptocurrencies, there needs to be a way to compare deposits in one versus the other, so as to fairly allocate proposal rights, determine whether or not a 2/3 threshold was passed, etc. this requires some form of price oracle. this can be done in a decentralized way (eg. see uniswap), but it introduces another component that could be manipulated and attacked by validators. pathological cryptocurrencies: one can always create a cryptocurrency that is pathologically constructed to nullify the impact of penalties. for example, one can imagine a fiat-backed token where coins that are seized by the protocol as penalties are tracked and not honored by the issuer, and the penalized actor's original balance is honored instead. this logic could even be implemented in a smart contract, and it's impossible to determine with certainty whether or not a given currency has such a mechanism built-in. reduced incentive alignment: if currencies other than the protocol's base token can be used to stake, this reduces the stakers' interest in seeing the protocol continue to operate and succeed. further reading https://github.com/ethereum/wiki/wiki/casper-proof-of-stake-compendium dark mode toggle hard forks, soft forks, defaults and coercion 2017 mar 14 see all posts one of the important arguments in the blockchain space is that of whether hard forks or soft forks are the preferred protocol upgrade mechanism. the basic difference between the two is that soft forks change the rules of a protocol by strictly reducing the set of transactions that is valid, so nodes following the old rules will still get on the new chain (provided that the majority of miners/validators implements the fork), whereas hard forks allow previously invalid transactions and blocks to become valid, so clients must upgrade their clients in order to stay on the hard-forked chain. there are also two sub-types of hard forks: strictly expanding hard forks, which strictly expand the set of transactions that is valid, and so effectively the old rules are a soft fork with respect to the new rules, and bilateral hard forks, where the two rulesets are incompatible both ways. here is a venn diagram to illustrate the fork types: the benefits commonly cited for the two are as follows. hard forks allow the developers much more flexibility in making the protocol upgrade, as they do not have to take care to make sure that the new rules "fit into" the old rules soft forks are more convenient for users, as users do not need to upgrade to stay on the chain soft forks are less likely to lead to a chain split soft forks only really require consent from miners/validators (as even if users still use the old rules, if the nodes making the chain use the new rules then only things valid under the new rules will get into the chain in any case); hard forks require opt-in consent from users aside from this, one major criticism often given for hard forks is that hard forks are "coercive". the kind of coercion implied here is not physical force; rather, it's coercion through network effect. that is, if the network changes rules from a to b, then even if you personally like a, if most other users like b and switch to b then you have to switch to b despite your personal disapproval of the change in order to be on the same network as everyone else. proponents of hard forks are often derided as trying to effect a "hostile take over" of a network, and "force" users to go along with them. additionally, the risk of chain splits is often used to bill hard forks as "unsafe". it is my personal viewpoint that these criticisms are wrong, and furthermore in many cases completely backwards. this viewpoint is not specific to ethereum, or bitcoin, or any other blockchain; it arises out of general properties of these systems, and is applicable to any of them. furthermore, the arguments below only apply to controversial changes, where a large portion of at least one constituency (miners/validators and users) disapprove of them; if a change is non-contentious, then it can generally be done safely no matter what the format of the fork is. first of all, let us discuss the question of coercion. hard forks and soft forks both change the protocol in ways that some users may not like; any protocol change will do this if it has less than exactly 100% support. furthermore, it is almost inevitable that at least some of the dissenters, in any scenario, value the network effect of sticking with the larger group more than they value their own preferences regarding the protocol rules. hence, both fork types are coercive, in the network-effect sense of the word. however, there is an essential difference between hard forks and soft forks: hard forks are opt-in, whereas soft forks allow users no "opting" at all. in order for a user to join a hard forked chain, they must personally install the software package that implements the fork rules, and the set of users that disagrees with a rule change even more strongly than they value network effects can theoretically simply stay on the old chain and, practically speaking, such an event has already happened. this is true in the case of both strictly expanding hard forks and bilateral hard forks. in the case of soft forks, however, if the fork succeeds the unforked chain does not exist. hence, soft forks clearly institutionally favor coercion over secession, whereas hard forks have the opposite bias. my own moral views lead me to favor secession over coercion, though others may differ (the most common argument raised is that network effects are really really important and it is essential that "one coin rule them all", though more moderate versions of this also exist). if i had to guess why, despite these arguments, soft forks are often billed as "less coercive" than hard forks, i would say that it is because it feels like a hard fork "forces" the user into installing a software update, whereas with a soft fork users do not "have" to do anything at all. however, this intuition is misguided: what matters is not whether or not individual users have to perform the simple bureaucratic step of clicking a "download" button, but rather whether or not the user is coerced into accepting a change in protocol rules that they would rather not accept. and by this metric, as mentioned above, both kinds of forks are ultimately coercive, and it is hard forks that come out as being somewhat better at preserving user freedom. now, let's look at highly controversial forks, particularly forks where miner/validator preferences and user preferences conflict. there are three cases here: (i) bilateral hard forks, (ii) strictly expanding hard forks, and (iii) so-called "user-activated soft forks" (uasf). a fourth category is where miners activate a soft fork without user consent; we will get to this later. first, bilateral hard forks. in the best case, the situation is simple. the two coins trade on the market, and traders decide the relative value of the two. from the etc/eth case, we have overwhelming evidence that miners are overwhelmingly likely to simply assign their hashrate to coins based on the ratio of prices in order to maximize their profit, regardless of their own ideological views. even if some miners profess ideological preferences toward one side or the other, it is overwhemingly likely that there will be enough miners that are willing to arbitrage any mismatch between price ratio and hashpower ratio, and bring the two into alignment. if a cartel of miners tries to form to not mine on one chain, there are overwheming incentives to defect. there are two edge cases here. the first is the possibilty that, because of an inefficient difficulty adjustment algorithm, the value of mining the coin goes down becase price drops but difficulty does not go down to compensate, making mining very unprofitable, and there are no miners willing to mine at a loss to keep pushing the chain forward until its difficulty comes back into balance. this was not the case with ethereum, but may well be the case with bitcoin. hence, the minority chain may well simply never get off the ground, and so it will die. note that the normative question of whether or not this is a good thing depends on your views on coercion versus secession; as you can imagine from what i wrote above i personally believe that such minority-chain-hostile difficulty adjustment algorithms are bad. the second edge case is that if the disparity is very large, the large chain can 51% attack the smaller chain. even in the case of an eth/etc split with a 10:1 ratio, this has not happened; so it is certainly not a given. however, it is always a possibility if miners on the dominant chain prefer coercion to allowing secession and act on these values. next, let's look at strictly expanding hard forks. in an sehf, there is the property that the non-forked chain is valid under the forked rules, and so if the fork has a lower price than the non-forked chain, it will have less hashpower than the non-forked chain, and so the non-forked chain will end up being accepted as the longest chain by both original-client and forked-client rules and so the forked chain "will be annihilated". there is an argument that there is thus a strong inherent bias against such a fork succeeding, as the possibility that the forked chain will get annihiliated will be baked into the price, pushing the price lower, making it even more likely that the chain will be annihilated... this argument to me seems strong, and so it is a very good reason to make any contentious hard fork bilateral rather than strictly expanding. bitcoin unlimited developers suggest dealing with this problem by making the hard fork bilateral manually after it happens, but a better choice would be to make the bilaterality built-in; for example, in the bitcoin case, one can add a rule to ban some unused opcode, and then make a transaction containing that opcode on the non-forked chain, so that under the forked rules the non-forked chain will from then on be considered forever invalid. in the ethereum case, because of various details about how state calculation works, nearly all hard forks are bilateral almost automatically. other chains may have different properties depending on their architecture. the last type of fork that was mentioned above is the user-activated soft fork. in a uasf, users turn on the soft fork rules without bothering to get consensus from miners; miners are expected to simply fall in line out of economic interest. if many users do not go along with the uasf, then there will be a coin split, and this will lead to a scenario identical to the strictly expanding hard fork, except and this is the really clever and devious part of the concept the same "risk of annihilation" pressure that strongly disfavors the forked chain in a strictly expanding hard fork instead strongly favors the forked chain in a uasf. even though a uasf is opt-in, it uses economic asymmetry in order to bias itself toward success (though the bias is not absolute; if a uasf is decidedly unpopular then it will not succeed and will simply lead to a chain split). however, uasfs are a dangerous game. for example, let us suppose that the developers of a project want to make a uasf patch that converts an unused opcode that previously accepted all transactions into an opcode that only accepts transactions that comply with the rules of some cool new feature, though one that is politically or technically controversial and miners dislike. miners have a clever and devious way to fight back: they can unilaterally implement a miner-activated soft fork that makes all transactions using the feature created by the soft fork always fail. now, we have three rulesets: the original rules where opcode x is always valid. the rules where opcode x is only valid if the rest of the transaction complies with the new rules the rules where opcode x is always invalid. note that (2) is a soft-fork with respect to (1), and (3) is a soft-fork with respect to (2). now, there is strong economic pressure in favor of (3), and so the soft-fork fails to accomplish its objective. the conclusion is this. soft forks are a dangerous game, and they become even more dangerous if they are contentious and miners start fighting back. strictly expanding hard forks are also a dangerous game. miner-activated soft forks are coercive; user-activated soft forks are less coercive, though still quite coercive because of the economic pressure, and they also have their dangers. if you really want to make a contentious change, and have decided that the high social costs of doing so are worth it, just do a clean bilateral hard fork, spend some time to add some proper replay protection, and let the market sort it out. on mining | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search on mining posted by vitalik buterin on june 19, 2014 research & development decentralization, n. the security assumption that a nineteen year old in hangzhou and someone who is maybe in the uk, and maybe not, have not yet decided to collude with each other. there has been a large amount of ruckus in the past week about the issue of mining centralization in the bitcoin network. we saw a single mining pool, ghash.io, amass over 45% hashpower for many hours, and at one point even grow to become 51% of the entire network. the entire front page of the bitcoin reddit was ablaze in intense discussion and a rare clash of complacency and fear, miners quickly mobilized to take their hashpower off ghash, and surprisingly clever strategies were used in an attempt to bring back the balance between the different pools, up to and including one miner with "between 50 th/s and 2 ph/s" mining at ghash but refusing to forward valid blocks, essentially sabotaging all mines on the pool to the extent of up to 4%. now, the situation has somewhat subsided, with ghash down to 35% network hashpower and the runner up, discus fish, up to 16%, and it is likely that the situation will remain that way for at least a short while before things heat up again. is the problem solved? of course not. can the problem be solved? that will be the primary subject of this post. bitcoin mining first of all, let us understand the problem. the purpose of bitcoin mining is to create a decentralized timestamping system, using what is essentially a majority vote mechanism to determine in which order certain transactions came as a way of solving the double-spending problem. the double-spending problem is simple to explain: if i send a transaction sending my 100 btc to you, and then one day later i send a transaction sending the same 100 btc to myself, both of those transactions obviously cannot simultaneously process. hence, one of the two has to "win", and the intuitively correct transaction that should get that honor is the one that came first. however, there is no way to look at a transaction and cryptographically determine when it was created. this is where bitcoin mining steps in. bitcoin mining works by having nodes called "miners" aggregate recent transactions and produce packages called "blocks". for a block to be valid, all of the transactions it contains must be valid, it must "point to" (ie. contain the hash of) a previous block that is valid, and it must satisfy "the proof of work condition" (namely, sha2562(block_header) <= 2190, ie. the double-hash of the block header must start with a large number of zeroes). because sha256 is a pseudorandom function, the only way to make such blocks is to repeatedly attempt to produce them until one happens to satisfy the condition. the 2190 "target" is a flexible parameter; it auto-adjusts so that on average the entire network needs to work for ten minutes before one node gets lucky and succeeds; once that happens, the newly produced block becomes the "latest" block, and everyone starts trying to mine a block pointing to that block as the previous block. this process, repeating once every ten minutes, constitutes the primary operation of the bitcoin network, creating an ever-lengthening chain of blocks ("blockchain") containing, in order, all of the transactions that have ever taken place. if a node sees two or more competing chains, it deems the one that is longest, ie. the one that has the most proof-of-work behind it, to be valid. over time, if two or more chains are simultaneously at play, one can see how the chain with more computational power backing it is eventually guaranteed to win; hence, the system can be described as "one cpu cycle, one vote". but there is one vulnerability: if one party, or one colluding group of parties, has over 50% of all network power, then that entity alone has majority control over the voting process and can out-compute any other chain. this gives this entity some privileges: the entity can only acknowledge blocks produced by itself as valid, preventing anyone else from mining because its own chain will always be the longest. over time, this doubles the miner's btc-denominated revenue at everyone else's expense. note that a weak version of this attack, "selfish-mining", starts to become effective at around 25% network power. the entity can refuse to include certain transactions (ie. censorship) the entity can "go back in time" and start mining from n blocks ago. when this fork inevitably overtakes the original, this removes the effect of any transactions that happened in the original chain after the forking point. this can be used to earn an illicit profit by (1) sending btc to an exchange, (2) waiting 6 blocks for the deposit to be confirmed, (3) purchasing and withdrawing ltc, (4) reversing the deposit transaction and instead sending those coins back to the attacker. this is the dreaded "51% attack". notably, however, even 99% hashpower does not give the attacker the privilege of assigning themselves an arbitrary number of new coins or stealing anyone else's coins (except by reversing transactions). another important point is that 51% of the network is not needed to launch such attacks; if all you want is to defraud a merchant who accepts transactions after waiting n confirmations (usually, n = 3 or n = 6 ), if your mining pool has portion p of the network you can succeed with probability (p / (1-p))^n ; at 35% hashpower and 3 confirmations, this means that ghash can currently steal altcoins from an altcoin exchange with 15.6% success probability once in every six tries. pools here is we get to pools. bitcoin mining is a rewarding but, unfortuantely, very high-variance activity. if, in the current 100 ph/s network, you are running an asic with 1 th/s, then every block you have a chance of 1 in 100000 of receiving the block reward of 25 btc, but the other 99999 times out of 100000 you get exactly nothing. given that network hashpower is currently doubling every three months (for simplicity, say 12500 blocks), that gives you a probability of 15.9% that your asic will ever generate a reward, and a 84.1% chance that the asic's total lifetime earnings will be exactly nothing. a mining pool acts as a sort of inverse insurance agent: the mining pool asks you to mine into into its own address instead of yours, and if you generate a block whose proof of work is almost good enough but not quite, called a "share", then the pool gives you a smaller payment. for example, if the mining difficulty for the main chain requires the hash to be less than 2190, then the requirement for a share might be 2190. hence, in this case, you will generate a share roughly every hundred blocks, receiving 0.024 btc from the pool, and one time in a thousand out of those the mining pool will receive a reward of 25 btc. the difference between the expected 0.00024 btc and 0.00025 btc per block is the mining pool's profit. however, mining pools also serve another purpose. right now, most mining asics are powerful at hashing, but surprisingly weak at everything else; the only thing they often have for general computation is a small raspberry pi, far too weak to download and validate the entire blockchain. miners could fix this, at the cost of something like an extra 100perdeviceforamoredecentcpu,buttheydonot−fortheobviousreasonthat100 per device for a more decent cpu, but they do not for the obvious reason that 100perdeviceforamoredecentcpu,buttheydonot−fortheobviousreasonthat0 is less than $100. instead, they ask mining pools to generate mining data for them. the "mining data" in question refers to the block header, a few hundred bytes of data containing the hash of the previous block, the root of a merkle tree containing transactions, the timestamp and some other ancillary data. miners take this data, and continue incrementing a value called a "nonce" until the block header satisfies the proof-of-work condition. ordinarily, miners would take this data from the block that they independently determine to be the latest block; here, however, the actual selection of what the latest block is is being relegated to the pools. thus, what do we have? well, right now, essentially this: the mining ecosystem has solidified into a relatively small number of pools, and each one has a substantial portion of the network and, of course, last week one of those pools, ghash, reached 51%. given that every time any mining pool, whether deepbit in 2011 or ghash in 2013, reached 51% there has been a sudden massive reduction in the number of users, it is entirely possible that ghash actually got anywhere up to 60% network hashpower, and is simply hiding some of it. there is plenty of evidence in the real world of large corporations creating supposedly mutually competing brands to give the appearance of choice and market dynamism, so such a hypothesis should not at all be discounted. even assuming that ghash is in fact being honest about the level of hashpower that it has, what this chart literally says is that the only reason why there are not 51% attacks happening against bitcoin right now is that discus fish, a mining pool run by a nineteen-year-old in hangzhou, china, and ghash, a mining pool run supposedly in the uk but may well be anywhere, have not yet decided to collude with each other and take over the blockchain. alternatively, if one is inclined to trust this particular nineteen-year-old in hangzhou (after all, he seemed quite nice when i met him), eligius or btcguild can collude with ghash instead. so what if, for the sake of example, ghash gets over 51% again and starts launching 51% attacks (or, perhaps, even starts launching attacks against altcoin exchanges at 40%)? what happens then? first of all, let us get one bad argument out of the way. some argue that it does not matter if ghash gets over 51%, because there is no incentive for them to perform attacks against the network since even one such attack would destroy the value of their own currency units and mining hardware. unfortunately, this argument is simply absurd. to see why, consider a hypothetical currency where the mining algorithm is simply a signature verifier for my own public key. only i can sign blocks, and i have every incentive to maintain trust in the system. why would the bitcoin community not adopt my clearly superior, non-electricity-wasteful, proof of work? there are many answers: i might be irrational, i might get coerced by a government, i might start slowly inculcating a culture where transaction reversals for certain "good purposes" (eg. blocking child pornography payments) are acceptable and then slowly expand that to cover all of my moral prejudices, or i might even have a massive short against bitcoin at 10x leverage. those middle two arguments are not crazy hypotheticals; they are real-world documented actions of the implemenation of me-coin that already exists: paypal. this is why decentralization matters; we do not burn millions of dollars of electricity per year just to move to a currency whose continued stability hinges on simply a slightly different kind of political game. additionally, it is important to note that even ghash itself has a history of involvement in using transaction reversal attacks against gambling sites; specifically, one may recall the episode involving betcoin dice. of course, ghash denies that it took any deliberate action, and is probably correct; rather, the attacks seem to be the fault of a rogue employee. however, this is not an argument in favor of ghash; much the opposite, it is a piece of real-world empirical evidence showing a common argument in favor of decentralization: power corrupts, and equally importantly power attracts those who are already corrupt. theoretically, ghash has increased security since then; in practice, no matter what they do this central point of vulnerability for the bitcoin network still exists. however, there is another, better, argument for why mining pools are not an issue: namely, precisely the fact that they are not individual miners, but rather pools from which miners can enter and leave at any time. because of this, one can reasonably say that ars technica's claim that bitcoin's security has been "shattered by an anonymous miner with 51% network power" is completely inaccurate; there is no one miner that controls anything close to 51%. there is indeed a single entity, called cex.io, that controls 25% of ghash, which is scary in itself but nevertheless far from the scenario that the headline is insinuating is the case. if individuals miners do not want to participate in subverting the bitcoin protocol and inevitably knocking the value of their coins down by something like 70%, they can simply leave the pool, and such a thing has now happened three times in bitcoin's history. however, the question is, as the bitcoin economy continues to professionalize, will this continue to be the case? or, given somewhat more "greedy" individuals, will the miners keep on mining at the only pool that lets them continue earning revenue, individually saving their own profits at the cost of taking the entire bitcoin mining ecosystem collectively down a cliff? solutions even now, there is actually one strategy that miners can, and have, taken to subvert ghash.io: mining on the pool but deliberately withholding any blocks they find that are actually valid. such a strategy is undetectable, but with a 1 ph/s miner mining in this way it essentially reduces the profits of all ghash miners by about 2.5%. this sort of pool sabotage completely negates the benefit of using the zero-fee ghash over other pools. this ability to punish bad actors is interesting, though its implications are unclear; what if ghash starts hiring miners to do the same against every other pool? thus, rather than relying on vigilante sabotage tactics with an unexamined economic endgame, we should ideally try to look for other solutions. first of all, there is the ever-present p2p mining pool, p2pool. p2pool has been around for years, and works by having its own internal blockchain with a 10-second block time, allowing miners to submit shares as blocks in the chain and requiring miners to attempt to produce blocks sending to all of the last few dozen share producers at the same time. if p2pool had 90% network hashpower, the result would not be centralization and benevolent dictatorship; rather, the limiting case would simply be a replica of the plain old bitcoin blockchain. however, p2pool has a problem: it requires miners to be fully validating nodes. as described above, given the possibility of mining without being a fully validating node this is unacceptable. one solution to this problem, and the solution that ethereum is taking, is to have a mining algorithm that forces nodes to store the entire blockchain locally. a simple algorithm for this in bitcoin's case is: def mine(block_header, n, nonce): o = [] for i in range(20): o.append(sha256(block_header + nonce + i)) n = [] for i in range(20): b = (o[i] / 2**128) % n n.append(tx(b, o[i])) return sha256(block_header + str(n)) where tx(b, k) is a function that returns the kth transaction in block b, wrapping around modulo the number of transactions in that block if necessary, and n is the current block number. note that this is a simple algorithm and is highly suboptimal; some obvious optimizations include making it serial (ie. o[i+1] depends on n[i]), building a merkle tree out of the o[i] values to allow them to be individually verified, and, as the ethereum protocol already does, maintaining a separate state tree and transaction list so the algorithm only needs to query the current block. the one minor roadblock to successful implementation in ethereum is simply that the current trie implementation has no concept of the "kth node" of a tree; the closest analog would probably need to be "first node with a key lexicographically after k, with wraparound", something for which it is possible to produce a compact patricia tree proof. this actually solves two problems at the same time. first, it removes the incentive to use a centralized pool instead of p2pool. second, there is an ongoing crisis in bitcoin about how there are too few full nodes; the reason why this is the case is that maintaining a full node with its 20gb blockchain is expensive, and no one wants to do it. with this scheme, every single mining asic would be forced to store the entire blockchain, a state from which performing all of the functions of a full node becomes trivial. a second strategy is another cryptographic trick: make mining non-outsourceable. specificically, the idea is to create a mining algorithm such that, when a miner creates a valid block, they always necessarily have an alternative way of publishing the block that secures the mining reward for themselves. the strategy is to use a cryptographic construction called a zero-knowledge proof, cryptographically proving that they created a valid block but keeping the block data secret, and then simultaneously create a block without proof of work that sends the reward to the miner. this would make it trivial to defraud a mining pool, making mining pools non-viable. such a setup would require a substantial change to bitcoin's mining algorithm, and uses cryptographic primitives far more advanced than those in the rest of bitcoin; arguably, complexity is in itself a serious disadvantage, and one that is perhaps worth it to solve serious problems like scalability but not to implement a clever trick to discourage mining pools. additionally, making mining pools impossible will arguably make the problem worse, not better. the reason why mining pools exist is to deal with the problem of variance; miners are not willing to purchase an investment which has only a 15% chance of earning any return. if the possibility of pooling is impossible, the mining economy will simply centralize into a smaller set of larger players a setup which, unlike now, individual participants cannot simply switch away from. the previous scheme, on the other hand, still allows pooling as long as the local node has the full blockchain, and thereby encourages a kind of pooling (namely, p2pool) that is not systemically harmful. another approach is less radical: don't change the mining algorithm at all, but change the pooling algorithms. right now, most mining pools use a payout scheme called "pay-per-last-n-shares" (pplns) pay miners per share an amount based on the revenue received from the last few thousand shares. this algorithm essentially splits the pool's own variance among its users, resulting in no risk for the pool and a small amount of variance for the users (eg. using a pool with 1% hashpower, the expected standard deviation of monthly returns is ~15%, far better than the solo mining lottery but still non-negligible). larger pools have less variance, because they mine more blocks (by basic statistics, a pool with 4x more mining power has a 2x smaller standard deviation as a percentage). there is another scheme, called pps (pay-per-share), where a mining pool simply pays a static amount per share to miners; this scheme removes all variance from miners, but at the cost of introducing risk to the pool; that is why no mining pool does it. meni rosenfeld's multi-pps attempts to provide a solution. instead of mining into one pool, miners can attempt to produce blocks which pay to many pools simultaneously (eg. 5 btc to one pool, 7 btc to another, 11.5 btc to a third and 1.5 btc to a fourth), and the pools will pay the miner for shares proportionately (eg. instead of one pool paying 0.024 btc per share, the first pool will pay 0.0048, the second 0.00672, the third 0.01104 and the fourth 0.00144). this allows very small pools to only accept miners giving them very small rewards, allowing them to take on a level of risk proportionate to their economic capabilities. for example, if pool a is 10x bigger than pool b, then pool a might accept blocks with outputs to them up to 10 btc, and pool b might only accept 1 btc. if one does the calculations, one can see that the expected return for pool b is exactly ten times what pool a gets in every circumstance, so pool b has no special superlinear advantage. in a single-pps scenario, on the other hand, the smaller b would face 3.16x higher risk compared to its wealth. the problem is, to what extent is the problem really because of variance, and to what extent is it something else, like convenience? sure, a 1% mining pool will see a 15% monthly standard deviation in its returns. however, all mining pools see something like a 40% monthly standard deviation in their returns simply because of the volatile btc price. the difference between 15% standard deviation and 2% standard deviation seems large and a compelling reason to use the largest pool; the difference between 42% and 55% not so much. so what other factors might influence mining pool centralization? another factor is the fact that pools necessarily "hear" about their own blocks instantly and everyone else's blocks after some network delay, so larger pools will be mining on outdated blocks less often; this problem is critical for blockchains with a time of ten seconds, but in bitcoin the effect is less than 1% and thus insignificant. a third factor is convenience; this can best be solved by funding an easy-to-use open-source make-your-own mining pool solution, in a similar spirit to the software used by many small vps providers; if deemed important, we may end up partially funding a network-agnostic version of such an effort. the last factor that still remains, however, is that ghash has no fee; rather, the pool sustains itself through its connection to the asic cloud-mining company cex.io, which controls 25% of its hashpower. thus, if we want to really get down to the bottom of the centralization problem, we may need to look at asics themselves. asics originally, bitcoin mining was intended to be a very egalitarian pursuit. millions of users around the world would all mine bitcoin on their desktops, and the result would be simultaneously a distribution model that is highly egalitarian and widely spreads out the initial btc supply and a consensus model that includes thousands of stakeholders, virtually precluding any possibility of collusion. initially, the scheme worked, ensuring that the first few million bitcoins got widely spread among many thousands of users, including even the normally cash-poor high school students. in 2010, however, came the advent of mining software for the gpu ("graphics processing unit"), taking advantage of the gpu's massive parallelization to achieve 10-100x speedups and rendering cpu mining completely unprofitable within months. in 2013, specialization took a further turn with the advent of asics. asics, or application-specific integrated circuits, are specialized mining chips produced with a single purpose: to crank out as many sha256 computations as possible in order to mine bitcoin blocks. as a result of this specialization, asics get a further 10-100x speedup over gpus, rendering gpu mining unprofitable as well. now, the only way to mine is to either start an asic company or purchase an asic from an existing one. the way the asic companies work is simple. first, the company starts up, does some minimal amount of setup work and figures out its plan, and starts taking preorders. these preorders are then used to fund the development of the asic, and once the asics are ready the devices are shipped to users, and the company starts manufacturing and selling more at a regular pace. asic manufacturing is done in a pipeline; there is one type of factory which produces the chips for asics, and then another, less sophisticated, operation, where the chips, together with standard parts like circuit boards and fans, are put together into complete boxes to be shipped to purchasers. so where does this leave us? it's obvious that asic production is fairly centralized; there are something like 10-30 companies manufacturing these devices, and each of them have a significant level of hashpower. however, i did not realize just how centralized asic production is until i visited this unassuming little building in shenzhen, china: on the third floor of the factory, we see: what we have in the first picture are about 150 miners of 780 gh/s each, making up a total 120 th/s of miners more than 0.1% of total network hashpower all in one place. the second picture shows boxes containing another 150 th/s. altogether, the factory produces slightly more than the sum of these two amounts about 300 th/s every single day. now, look at this chart: in total, the bitcoin network gains about 800 th/s every day. thus, even adding some safety factors and assuming the factory shuts down some days a week, what we have is one single factory producing over a quarter of all new hashpower being added to the bitcoin network. now, the building is a bit large, so guess what's on the first floor? that's right, a fabrication facility producing scrypt asics equal to a quarter of all new hashpower added to the litecoin network. this projects an image of a frightening endgame for bitcoin: the bitcoin network spending millions of dollars of electricity every year only to replace the us dollar's mining algorithm of "8 white guys" with a few dozen guys in shenzhen. however, before we get too alarmist about the future of mining, it is important to dig down and understand (1) what's wrong with asics, (2) what's okay with cpus, and (3) what the future of asic mining is going to look like. the question is a more complex one than it seems. first of all, one might ask, why is it bad that asics are only produced by a few companies and a quarter of them pass through one factory? cpus are also highly centralized; integrated circuits are being produced by only a small number of companies, and nearly all computers that we use have at least some components from amd or intel. the answer is, although amd and intel produce the cpus, they do not control what's run on them. they are general-purpose devices, and there is no way for the manufacturers to translate their control over the manufacturing process into any kind of control over its use. drm-laden "trusted computing modules" do exist, but it is very difficult to imagine such a thing being used to force a computer to participate in a double-spend attack. with asic miners, right now things are still not too bad. although asics are produced in only a small number of factories, they are still controlled by thousands of people worldwide in disparate data centers and homes, and individual miners each usually with less than a few terahashes have the ability to direct their hashpower wherever they need. soon, however, that may change. in a month's time, what if the manufacturers realize that it does not make economic sense for them to sell their asics when they can instead simply keep all of their devices in a central warehouse and earn the full revenue? shipping costs would drop to near-zero, shipping delays would go down (one week shipping delay corresponds to ~5.6% revenue loss at current hashpower growth rates) and there would be no need to produce stable or pretty casings. in that scenario, it would not just be 25% of all asics that are produced by one factory in shenzhen; it would be 25% of all hashpower run out of one factory in shenzhen. when visiting the headquarters of a company in hangzhou that is involved, among other things, in litecoin mining, i asked the founders the same question: why don't you just keep miners in-house? they provided three answers. first, they care about decentralization. this is simple to understand, and is very fortunate that so many miners feel this way for the time being, but ultimately mining will be carried out by firms that care a little more about monetary profit and less about ideology. second, they need pre-orders to fund the company. reasonable, but solvable by issuing "mining contracts" (essentially, crypto-assets which pay out dividends equal to a specific number of gh/s of mining power). third, there's not enough electricity and space in the warehouses. the last argument, as specious as it seems, may be the only one to hold water in the long term; it is also the stated reason why asicminer stopped mining purely in-house and started selling usb miners to consumers, suggesting that perhaps there is a strong and universal rationale behind such a decision. assuming that the funding strategies of selling pre-orders and selling mining contracts are economically equivalent (which they are), the equation for determining whether in-house mining or selling makes more sense is as follows: on the left side, we have the costs of in-house mining: electricity, storage and maintenance. on the right side, we have the cost of electricity, storage and maintenance externally (ie. in buyers' hands), shipping and the penalty from having to start running the asic later, as well as a negative factor to account for the fact that some people mine at least partially for fun and out of an ideological desire to support the network. let's analyze these figures right now. we'll use the butterfly labs monarch as our example, and keep each asic running for one year for simplicity. internal electricity, storage, maintenance according to bfl's checkout page, internal electricity, storage and maintennance cost $1512 per year, which we will mark down to $1000 assuming bfl takes some profit external electricity in ontario, prices are about $0.1 per kwh. a butterfly labs monarch will run 600 gh/s at 350 w; normalizing this to per-th, this means an electricity cost of $1.40 per day or $511 for the entire year external storage at home, one can consider storage free, or one can add a convenience fee of $1 per day; hence, we'll say somewhere from $0 to $365 external maintenance hard to quantify this value; for technically skilled invididuals who enjoy the challenge it's zero, and for others it might be hard; hence, we can say $0 to $730 shipping cost according to bfl, $38. revenue currently, 1 th/s gives you 0.036 btc or $21.6 per day. since in our analysis hashpower doubles every 90 days, so the effectiveness of the asic halves every 90 days, we get 122 days of life or $2562 revenue shipping time according to my chinese sources, one week hashpower doubling time three months. hence, the entire expression for the shipping delay penalty is 2562 * (1 0.5 ^ 0.0769) = 133.02 hobbyist/ideology premium currently, a large portion of bitcoin miners are doing it out of ideological considerations, so we can say anywhere from $0 to $1000 thus, adding it all up, on the left we have $1000, and on the right we have $511 + $38 + $133 = $682, up to plus $1095 and minus up to $1000. thus, it's entirely ambiguous which one is better; errors in my analysis and the nebulous variables of how much people value their time and aesthetics seem to far outweigh any definite conclusions. but what will happen in the future? fundamentally, one can expect that electricity, storage and maintenance would be much cheaper centrally than with each consumer simply due to economies of scale and gains from specialization; additionally most people in the "real world" are not altruists, hobbyists or admirers of beautiful asic coverings. shipping cost are above zero, and the shipping delay penalty is above zero. so thus it seems that the economics roundly favor centralized mining... ... except for one potential factor: heat. right now, asics are still in a rapid development phase, so the vast majority of the cost is hardware; the bfl miner used in the above example costs 2200,buttheelectricitycosts2200, but the electricity costs 2200,buttheelectricitycosts511. in the future, however, development will be much slower; ultimately we can expect a convergence to moore's law, with hashpower doubling every two years, and even moore's law itself seems to be slowing. in such a world, electricity costs may come back as the primary choke point. but how much does electricity cost? in a centralized warehouse, quite a lot, and the square-cube law guarantees that in a centralized environment even more energy than at home would need to be spent on cooling because all of the miners are in one place and most of them are too deep inside the factory to have exposure to cool fresh air. in a home, however, if the outside temperature is less than about 20'c, the cost of electricity is zero; all electricity spent by the miner necessarily eventually turns into "waste" heat, which then heats the home and substitutes for electricity that would be spent by a central heater. this is the only argument for why asic decentralization may work: rather than decentralization happening because everyone has a certain quantity of unused, and thereby free, units of computational time on their laptop, decentralization happens because many people have a certain quantity of demand for heating in their homes. will this happen? many bitcoin proponents seem convinced that the answer is yes. however, i am not sure; it is an entirely empirical question whether or not electricity cost is less than maintenance plus storage plus shipping plus shipping delay penalty, and in ten years' time the equation may well fall on one side or the other. i personally am not willing to simply sit back and hope for the best. this is why i personally find it disappointing that so many of the core bitcoin developers (though fortunately not nearly all) are content to consider the proof of work problem "solved" or argue that attempting to solve mining specialization is an act of "needless re-engineering". it may prove to be, or it may not, but the fact that we are having this discussion in the first place strongly suggests that bitcoin's current approach is very far from perfect. asic resistance the solution to the asic problem that is most often touted is the development of asic-resistant mining algorithms. so far, there have been two lines of thought in developing such algorithms. the first is memory-hardness reducing the power of asics to achieve massive gains through parallelization by using a function which takes a very large amount of memory. the community's first attempt was scrypt, which proved to be not resistant enough; in january, i attempted to improve scrypt's memory-hardness with dagger, an algorithm which is memory-hard to compute (to the extent of 128 mb) but easy to verify; however, this algorithm is vulnerable to shared-memory attacks where a number of parallel processes can access the same 128 mb of memory. the current state-of-the-art in memory-hard pow is cuckoo, an algorithm which looks for length-42 cycles in graphs. it takes a large amount of memory to efficiently find such cycles, but a cycle is very quick to verify, requiring 42 hashes and less than 70 bytes of memory. the second approach is somewhat different: create a mechanism for generating new hash functions, and make the space of functions that it generates so large that the kind of computer best suited to processing them is by definition completely generalized, ie. a cpu. this approach gets close to being "provably asic resistant" and thus more future-proof, rather than focusing on specific aspects like memory, but it too is imperfect; there will always be at least some parts of a cpu that will prove to be extraneous in such an algorithm and can be removed for efficiency. however, the quest is not for perfect asic resistance; rather, the challenge is to achieve what we can call "economic asic resistance" building an asic should not be worth it. this is actually surprisingly likely to be achievable. to see why, note that mining output per dollar spent is, for most people, sublinear. the first n units of mining power are very cheap to produce, since users can simply use the existing unused computational time on their desktops and only pay for electricity (e). going beyond n units, however, one needs to pay for both hardware and electricity (h + e). if asics are feasible, as long as their speedup over commodity hardware is less than (h + e) / e, then even in an asic-containing ecosystem it will be profitable for people to spend their electricity mining on their desktops. this is the goal that we wish to strive for; whether we can reach it or not is entirely unknown, but since cryptocurrency as a whole is a massive experiment in any case it does not hurt to try. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements what is ethereum? ethereum is often described as "the world computer.” but what does that mean? let’s start with a computer science–focused description, and then try to decipher that with a more practical analysis of ethereum’s capabilities and characteristics, while comparing it to bitcoin and other decentralized information exchange platforms (or "blockchains" for short). from a computer science perspective, ethereum is a deterministic but practically unbounded state machine, consisting of a globally accessible singleton state and a virtual machine that applies changes to that state. from a more practical perspective, ethereum is an open source, globally decentralized computing infrastructure that executes programs called smart contracts. it uses a blockchain to synchronize and store the system’s state changes, along with a cryptocurrency called ether to meter and constrain execution resource costs. the ethereum platform enables developers to build powerful decentralized applications with built-in economic functions. while providing high availability, auditability, transparency, and neutrality, it also reduces or eliminates censorship and reduces certain counterparty risks. compared to bitcoin many people will come to ethereum with some prior experience of cryptocurrencies, specifically bitcoin. ethereum shares many common elements with other open blockchains: a peer-to-peer network connecting participants, a byzantine fault–tolerant consensus algorithm for synchronization of state updates (a proof-of-work blockchain), the use of cryptographic primitives such as digital signatures and hashes, and a digital currency (ether). yet in many ways, both the purpose and construction of ethereum are strikingly different from those of the open blockchains that preceded it, including bitcoin. ethereum’s purpose is not primarily to be a digital currency payment network. while the digital currency ether is both integral to and necessary for the operation of ethereum, ether is intended as a utility currency to pay for use of the ethereum platform as the world computer. unlike bitcoin, which has a very limited scripting language, ethereum is designed to be a general-purpose programmable blockchain that runs a virtual machine capable of executing code of arbitrary and unbounded complexity. where bitcoin’s script language is, intentionally, constrained to simple true/false evaluation of spending conditions, ethereum’s language is turing complete, meaning that ethereum can straightforwardly function as a general-purpose computer. components of a blockchain the components of an open, public blockchain are (usually): a peer-to-peer (p2p) network connecting participants and propagating transactions and blocks of verified transactions, based on a standardized "gossip" protocol messages, in the form of transactions, representing state transitions a set of consensus rules, governing what constitutes a transaction and what makes for a valid state transition a state machine that processes transactions according to the consensus rules a chain of cryptographically secured blocks that acts as a journal of all the verified and accepted state transitions a consensus algorithm that decentralizes control over the blockchain, by forcing participants to cooperate in the enforcement of the consensus rules a game-theoretically sound incentivization scheme (e.g., proof-of-work costs plus block rewards) to economically secure the state machine in an open environment one or more open source software implementations of the above ("clients") all or most of these components are usually combined in a single software client. for example, in bitcoin, the reference implementation is developed by the bitcoin core open source project and implemented as the bitcoind client. in ethereum, rather than a reference implementation there is a reference specification, a mathematical description of the system in the yellow paper (see further reading). there are a number of clients, which are built according to the reference specification. in the past, we used the term "blockchain" to represent all of the components just listed, as a shorthand reference to the combination of technologies that encompass all of the characteristics described. today, however, there are a huge variety of blockchains with different properties. we need qualifiers to help us understand the characteristics of the blockchain in question, such as open, public, global, decentralized, neutral, and censorship-resistant, to identify the important emergent characteristics of a "blockchain" system that these components allow. not all blockchains are created equal. when someone tells you that something is a blockchain, you have not received an answer; rather, you need to start asking a lot of questions to clarify what they mean when they use the word "blockchain." start by asking for a description of the components in the preceding list, then ask whether this "blockchain" exhibits the characteristics of being open, public, etc. the birth of ethereum all great innovations solve real problems, and ethereum is no exception. ethereum was conceived at a time when people recognized the power of the bitcoin model, and were trying to move beyond cryptocurrency applications. but developers faced a conundrum: they either needed to build on top of bitcoin or start a new blockchain. building upon bitcoin meant living within the intentional constraints of the network and trying to find workarounds. the limited set of transaction types, data types, and sizes of data storage seemed to limit the sorts of applications that could run directly on bitcoin; anything else needed additional off-chain layers, and that immediately negated many of the advantages of using a public blockchain. for projects that needed more freedom and flexibility while staying on-chain, a new blockchain was the only option. but that meant a lot of work: bootstrapping all the infrastructure elements, exhaustive testing, etc. toward the end of 2013, vitalik buterin, a young programmer and bitcoin enthusiast, started thinking about further extending the capabilities of bitcoin and mastercoin (an overlay protocol that extended bitcoin to offer rudimentary smart contracts). in october of that year, vitalik proposed a more generalized approach to the mastercoin team, one that allowed flexible and scriptable (but not turing-complete) contracts to replace the specialized contract language of mastercoin. while the mastercoin team were impressed, this proposal was too radical a change to fit into their development roadmap. in december 2013, vitalik started sharing a whitepaper that outlined the idea behind ethereum: a turing-complete, general-purpose blockchain. a few dozen people saw this early draft and offered feedback, helping vitalik evolve the proposal. both of the authors of this book received an early draft of the whitepaper and commented on it. andreas m. antonopoulos was intrigued by the idea and asked vitalik many questions about the use of a separate blockchain to enforce consensus rules on smart contract execution and the implications of a turing-complete language. andreas continued to follow ethereum’s progress with great interest but was in the early stages of writing his book mastering bitcoin, and did not participate directly in ethereum until much later. dr. gavin wood, however, was one of the first people to reach out to vitalik and offer to help with his c++ programming skills. gavin became ethereum’s cofounder, codesigner, and cto. as vitalik recounts in his "ethereum prehistory" post: this was the time when the ethereum protocol was entirely my own creation. from here on, however, new participants started to join the fold. by far the most prominent on the protocol side was gavin wood…​ gavin can also be largely credited for the subtle change in vision from viewing ethereum as a platform for building programmable money, with blockchain-based contracts that can hold digital assets and transfer them according to pre-set rules, to a general-purpose computing platform. this started with subtle changes in emphasis and terminology, and later this influence became stronger with the increasing emphasis on the “web 3” ensemble, which saw ethereum as being one piece of a suite of decentralized technologies, the other two being whisper and swarm. starting in december 2013, vitalik and gavin refined and evolved the idea, together building the protocol layer that became ethereum. ethereum’s founders were thinking about a blockchain without a specific purpose, that could support a broad variety of applications by being programmed. the idea was that by using a general-purpose blockchain like ethereum, a developer could program their particular application without having to implement the underlying mechanisms of peer-to-peer networks, blockchains, consensus algorithms, etc. the ethereum platform was designed to abstract these details and provide a deterministic and secure programming environment for decentralized blockchain applications. much like satoshi, vitalik and gavin didn’t just invent a new technology; they combined new inventions with existing technologies in a novel way and delivered the prototype code to prove their ideas to the world. the founders worked for years, building and refining the vision. and on july 30, 2015, the first ethereum block was mined. the world’s computer started serving the world. note vitalik buterin’s article "a prehistory of ethereum" was published in september 2017 and provides a fascinating first-person view of ethereum’s earliest moments. you can read it at https://vitalik.ca/general/2017/09/14/prehistory.html. ethereum’s four stages of development ethereum’s development was planned over four distinct stages, with major changes occurring at each stage. a stage may include subreleases, known as "hard forks," that change functionality in a way that is not backward compatible. the four main development stages are codenamed frontier, homestead, metropolis, and serenity. the intermediate hard forks that have occurred to date are codenamed ice age, dao, tangerine whistle, spurious dragon, byzantium, constantinople/st. petersburg, istanbul and muir glacier. both the development stages and the intermediate hard forks are shown on the following timeline, which is "dated" by block number: block #0 frontier—the initial stage of ethereum, lasting from july 30, 2015, to march 2016. block #200,000 ice age—a hard fork to introduce an exponential difficulty increase, to motivate a transition to pos when ready. block #1,150,000 homestead—the second stage of ethereum, launched in march 2016. block #1,192,000 dao—a hard fork that reimbursed victims of the hacked dao contract and caused ethereum and ethereum classic to split into two competing systems. block #2,463,000 tangerine whistle—a hard fork to change the gas calculation for certain i/o-heavy operations and to clear the accumulated state from a denial-of-service (dos) attack that exploited the low gas cost of those operations. block #2,675,000 spurious dragon—a hard fork to address more dos attack vectors, and another state clearing. also, a replay attack protection mechanism. block #4,370,000 metropolis byzantium—metropolis is the third stage of ethereum. launched in october 2017, byzantium is the first part of metropolis, adding low-level functionalities and adjusting the block reward and difficulty. block #7,280,000 constantinople / st. petersburg—constantinople was planned to be the second part of metropolis with similar improvements. a few hours before its activation, a critical bug was discovered. the hard fork was therefore postponed and renamed st. petersburg. block #9,069,000 istanbul—an additional hard fork with the same approach, and naming convention, as for the prior two. block #9,200,000 muir glacier—a hard fork whose sole purpose was to adjust the difficulty again due to the exponential increase introduced by ice age. two hard forks, berlin and london, have also been announced, and we are now in the final stage of ethereum development, codenamed serenity. serenity involves a profound reorganization of the infrastructure that will make ethereum more scalable, more secure, and more sustainable. it is presented as the second version of ethereum, "ethereum 2.0". ethereum: a general-purpose blockchain the original blockchain, namely bitcoin’s blockchain, tracks the state of units of bitcoin and their ownership. you can think of bitcoin as a distributed consensus state machine, where transactions cause a global state transition, altering the ownership of coins. the state transitions are constrained by the rules of consensus, allowing all participants to (eventually) converge on a common (consensus) state of the system, after several blocks are mined. ethereum is also a distributed state machine. but instead of tracking only the state of currency ownership, ethereum tracks the state transitions of a general-purpose data store, i.e., a store that can hold any data expressible as a key–value tuple. a key–value data store holds arbitrary values, each referenced by some key; for example, the value "mastering ethereum" referenced by the key "book title". in some ways, this serves the same purpose as the data storage model of random access memory (ram) used by most general-purpose computers. ethereum has memory that stores both code and data, and it uses the ethereum blockchain to track how this memory changes over time. like a general-purpose stored-program computer, ethereum can load code into its state machine and run that code, storing the resulting state changes in its blockchain. two of the critical differences from most general-purpose computers are that ethereum state changes are governed by the rules of consensus and the state is distributed globally. ethereum answers the question: "what if we could track any arbitrary state and program the state machine to create a world-wide computer operating under consensus?" ethereum’s components in ethereum, the components of a blockchain system described in components of a blockchain are, more specifically: p2p network ethereum runs on the ethereum main network, which is addressable on tcp port 30303, and runs a protocol called ðξvp2p. consensus rules ethereum’s consensus rules are defined in the reference specification, the yellow paper (see further reading). transactions ethereum transactions are network messages that include (among other things) a sender, recipient, value, and data payload. state machine ethereum state transitions are processed by the ethereum virtual machine (evm), a stack-based virtual machine that executes bytecode (machine-language instructions). evm programs, called "smart contracts," are written in high-level languages (e.g., solidity) and compiled to bytecode for execution on the evm. data structures ethereum’s state is stored locally on each node as a database (usually google’s leveldb), which contains the transactions and system state in a serialized hashed data structure called a merkle patricia tree. consensus algorithm ethereum uses bitcoin’s consensus model, nakamoto consensus, which uses sequential single-signature blocks, weighted in importance by pow to determine the longest chain and therefore the current state. however, there are plans to move to a pos weighted voting system, codenamed casper, in the near future. economic security ethereum currently uses a pow algorithm called ethash, but this will eventually be dropped with the move to pos at some point in the future. clients ethereum has several interoperable implementations of the client software, the most prominent of which are go-ethereum (geth) and parity. further reading the following references provide additional information on the technologies mentioned here: the ethereum yellow paper: https://ethereum.github.io/yellowpaper/paper.pdf the beige paper, a rewrite of the yellow paper for a broader audience in less formal language: https://github.com/chronaeon/beigepaper ðξvp2p network protocol: https://github.com/ethereum/devp2p/blob/master/rlpx.md ethereum virtual machine list of resources: https://eth.wiki/en/concepts/evm/ethereum-virtual-machine-(evm)-awesome-list leveldb database (used most often to store the local copy of the blockchain): https://github.com/google/leveldb merkle patricia trees: https://eth.wiki/en/fundamentals/patricia-tree ethash pow algorithm: https://eth.wiki/en/concepts/ethash/ethash casper pos v1 implementation guide: http://bit.ly/2dypr3l go-ethereum (geth) client: https://geth.ethereum.org/ parity ethereum client: https://parity.io/ ethereum and turing completeness as soon as you start reading about ethereum, you will immediately encounter the term "turing complete." ethereum, they say, unlike bitcoin, is turing complete. what exactly does that mean? the term refers to english mathematician alan turing, who is considered the father of computer science. in 1936 he created a mathematical model of a computer consisting of a state machine that manipulates symbols by reading and writing them on sequential memory (resembling an infinite-length paper tape). with this construct, turing went on to provide a mathematical foundation to answer (in the negative) questions about universal computability, meaning whether all problems are solvable. he proved that there are classes of problems that are uncomputable. specifically, he proved that the halting problem (whether it is possible, given an arbitrary program and its input, to determine whether the program will eventually stop running) is not solvable. alan turing further defined a system to be turing complete if it can be used to simulate any turing machine. such a system is called a universal turing machine (utm). ethereum’s ability to execute a stored program, in a state machine called the ethereum virtual machine, while reading and writing data to memory makes it a turing-complete system and therefore a utm. ethereum can compute any algorithm that can be computed by any turing machine, given the limitations of finite memory. ethereum’s groundbreaking innovation is to combine the general-purpose computing architecture of a stored-program computer with a decentralized blockchain, thereby creating a distributed single-state (singleton) world computer. ethereum programs run "everywhere," yet produce a common state that is secured by the rules of consensus. turing completeness as a "feature" hearing that ethereum is turing complete, you might arrive at the conclusion that this is a feature that is somehow lacking in a system that is turing incomplete. rather, it is the opposite. turing completeness is very easy to achieve; in fact, the simplest turing-complete state machine known has 4 states and uses 6 symbols, with a state definition that is only 22 instructions long. indeed, sometimes systems are found to be "accidentally turing complete." a fun reference of such systems can be found at http://bit.ly/2og1vgx. however, turing completeness is very dangerous, particularly in open access systems like public blockchains, because of the halting problem we touched on earlier. for example, modern printers are turing complete and can be given files to print that send them into a frozen state. the fact that ethereum is turing complete means that any program of any complexity can be computed by ethereum. but that flexibility brings some thorny security and resource management problems. an unresponsive printer can be turned off and turned back on again. that is not possible with a public blockchain. implications of turing completeness turing proved that you cannot predict whether a program will terminate by simulating it on a computer. in simple terms, we cannot predict the path of a program without running it. turing-complete systems can run in "infinite loops," a term used (in oversimplification) to describe a program that does not terminate. it is trivial to create a program that runs a loop that never ends. but unintended never-ending loops can arise without warning, due to complex interactions between the starting conditions and the code. in ethereum, this poses a challenge: every participating node (client) must validate every transaction, running any smart contracts it calls. but as turing proved, ethereum can’t predict if a smart contract will terminate, or how long it will run, without actually running it (possibly running forever). whether by accident or on purpose, a smart contract can be created such that it runs forever when a node attempts to validate it. this is effectively a dos attack. and of course, between a program that takes a millisecond to validate and one that runs forever are an infinite range of nasty, resource-hogging, memory-bloating, cpu-overheating programs that simply waste resources. in a world computer, a program that abuses resources gets to abuse the world’s resources. how does ethereum constrain the resources used by a smart contract if it cannot predict resource use in advance? to answer this challenge, ethereum introduces a metering mechanism called gas. as the evm executes a smart contract, it carefully accounts for every instruction (computation, data access, etc.). each instruction has a predetermined cost in units of gas. when a transaction triggers the execution of a smart contract, it must include an amount of gas that sets the upper limit of what can be consumed running the smart contract. the evm will terminate execution if the amount of gas consumed by computation exceeds the gas available in the transaction. gas is the mechanism ethereum uses to allow turing-complete computation while limiting the resources that any program can consume. the next question is, 'how does one get gas to pay for computation on the ethereum world computer?' you won’t find gas on any exchanges. it can only be purchased as part of a transaction, and can only be bought with ether. ether needs to be sent along with a transaction and it needs to be explicitly earmarked for the purchase of gas, along with an acceptable gas price. just like at the pump, the price of gas is not fixed. gas is purchased for the transaction, the computation is executed, and any unused gas is refunded back to the sender of the transaction. from general-purpose blockchains to decentralized applications (dapps) ethereum started as a way to make a general-purpose blockchain that could be programmed for a variety of uses. but very quickly, ethereum’s vision expanded to become a platform for programming dapps. dapps represent a broader perspective than smart contracts. a dapp is, at the very least, a smart contract and a web user interface. more broadly, a dapp is a web application that is built on top of open, decentralized, peer-to-peer infrastructure services. a dapp is composed of at least: smart contracts on a blockchain a web frontend user interface in addition, many dapps include other decentralized components, such as: a decentralized (p2p) storage protocol and platform a decentralized (p2p) messaging protocol and platform tip you may see dapps spelled as ðapps. the ð character is the latin character called "eth," alluding to ethereum. to display this character, use the unicode codepoint 0xd0, or if necessary the html character entity eth (or decimal entity #208). the third age of the internet in 2004 the term "web 2.0" came to prominence, describing an evolution of the web toward user-generated content, responsive interfaces, and interactivity. web 2.0 is not a technical specification, but rather a term describing the new focus of web applications. the concept of dapps is meant to take the world wide web to its next natural evolutionary stage, introducing decentralization with peer-to-peer protocols into every aspect of a web application. the term used to describe this evolution is web3, meaning the third "version" of the web. first proposed by dr. gavin wood, web3 represents a new vision and focus for web applications: from centrally owned and managed applications, to applications built on decentralized protocols. in later chapters we’ll explore the ethereum web3.js javascript library, which bridges javascript applications that run in your browser with the ethereum blockchain. the web3.js library also includes an interface to a p2p storage network called swarm and a p2p messaging service called whisper. with these three components included in a javascript library running in your web browser, developers have a full application development suite that allows them to build web3 dapps. ethereum’s development culture so far we’ve talked about how ethereum’s goals and technology differ from those of other blockchains that preceded it, like bitcoin. ethereum also has a very different development culture. in bitcoin, development is guided by conservative principles: all changes are carefully studied to ensure that none of the existing systems are disrupted. for the most part, changes are only implemented if they are backward compatible. existing clients are allowed to opt-in, but will continue to operate if they decide not to upgrade. in ethereum, by comparison, the community’s development culture is focused on the future rather than the past. the (not entirely serious) mantra is "move fast and break things." if a change is needed, it is implemented, even if that means invalidating prior assumptions, breaking compatibility, or forcing clients to update. ethereum’s development culture is characterized by rapid innovation, rapid evolution, and a willingness to deploy forward-looking improvements, even if this is at the expense of some backward compatibility. what this means to you as a developer is that you must remain flexible and be prepared to rebuild your infrastructure as some of the underlying assumptions change. one of the big challenges facing developers in ethereum is the inherent contradiction between deploying code to an immutable system and a development platform that is still evolving. you can’t simply "upgrade" your smart contracts. you must be prepared to deploy new ones, migrate users, apps, and funds, and start over. ironically, this also means that the goal of building systems with more autonomy and less centralized control is still not fully realized. autonomy and decentralization require a bit more stability in the platform than you’re likely to get in ethereum in the next few years. in order to "evolve" the platform, you have to be ready to scrap and restart your smart contracts, which means you have to retain a certain degree of control over them. but, on the positive side, ethereum is moving forward very fast. there’s little opportunity for "bike-shedding," an expression that means holding up development by arguing over minor details such as how to build the bicycle shed at the back of a nuclear power station. if you start bike-shedding, you might suddenly discover that while you were distracted the rest of the development team changed the plan and ditched bicycles in favor of autonomous hovercraft. eventually, the development of the ethereum platform will slow down and its interfaces will become fixed. but in the meantime, innovation is the driving principle. you’d better keep up, because no one will slow down for you. why learn ethereum? blockchains have a very steep learning curve, as they combine multiple disciplines into one domain: programming, information security, cryptography, economics, distributed systems, peer-to-peer networks, etc. ethereum makes this learning curve a lot less steep, so you can get started quickly. but just below the surface of a deceptively simple environment lies a lot more. as you learn and start looking deeper, there’s always another layer of complexity and wonder. ethereum is a great platform for learning about blockchains and it’s building a massive community of developers, faster than any other blockchain platform. more than any other, ethereum is a developer’s blockchain, built by developers for developers. a developer familiar with javascript applications can drop into ethereum and start producing working code very quickly. for the first few years of ethereum’s life, it was common to see t-shirts announcing that you can create a token in just five lines of code. of course, this is a double-edged sword. it’s easy to write code, but it’s very hard to write good and secure code. what this book will teach you this book dives into ethereum and examines every component. you will start with a simple transaction, dissect how it works, build a simple contract, make it better, and follow its journey through the ethereum system. you will learn not only how to use ethereum—how it works—but also why it is designed the way it is. you will be able to understand how each of the pieces works, and how they fit together and why. dark mode toggle a prehistory of the ethereum protocol 2017 sep 14 see all posts although the ideas behind the current ethereum protocol have largely been stable for two years, ethereum did not emerge all at once, in its current conception and fully formed. before the blockchain has launched, the protocol went through a number of significant evolutions and design decisions. the purpose of this article will be to go through the various evolutions that the protocol went through from start to launch; the countless work that was done on the implementations of the protocol such as geth, cppethereum, pyethereum, and ethereumj, as well as the history of applications and businesses in the ethereum ecosystem, is deliberately out of scope. also out of scope is the history of casper and sharding research. while we can certainly make more blog posts talking about all of the various ideas vlad, gavin, myself and others came up with, and discarded, including "proof of proof of work", hub-and-spoke chains, "hypercubes", shadow chains (arguably a precursor to plasma), chain fibers, and various iterations of casper, as well as vlad's rapidly evolving thoughts on reasoning about incentives of actors in consensus protocols and properties thereof, this would also be far too complex a story to go through in one post, so we will leave it out for now. let us first begin with the very earliest version of what would eventually become ethereum, back when it was not even called ethereum. when i was visiting israel in october 2013, i spent quite a bit of time with the mastercoin team, and even suggested a few features for them. after spending a couple of times thinking about what they were doing, i sent the team a proposal to make their protocol more generalized and support more types of contracts without adding an equally large and complex set of features: https://web.archive.org/web/20150627031414/http://vbuterin.com/ultimatescripting.html notice that this is very far from the later and more expansive vision of ethereum: it specialized purely in what mastercoin was trying to specialize in already, namely two-party contracts where parties a and b would both put in money, and then they would later get money out according to some formula specified in the contract (eg. a bet would say "if x happens then give all the money to a, otherwise give all the money to b"). the scripting language was not turing-complete. the mastercoin team was impressed, but they were not interested in dropping everything they were doing to go in this direction, which i was increasingly convinced is the correct choice. so here comes version 2, circa december: https://web.archive.org/web/20131219030753/http://vitalik.ca/ethereum.html here you can see the results of a substantial rearchitecting, largely a result of a long walk through san francisco i took in november once i realized that smart contracts could potentially be fully generalized. instead of the scripting language being simply a way of describing the terms of relations between two parties, contracts were themselves fully-fledged accounts, and had the ability to hold, send and receive assets, and even maintain a permanent storage (back then, the permanent storage was called "memory", and the only temporary "memory" was the 256 registers). the language switched from being a stack-based machine to being a register-based one on my own volition; i had little argument for this other than that it seemed more sophisticated. additionally, notice that there is now a built-in fee mechanism: at this point, ether literally was gas; after every single computational step, the balance of the contract that a transaction was calling would drop a little bit, and if the contract ran out of money execution would halt. note that this "receiver pays" mechanism meant that the contract itself had to require the sender to pay the contract a fee, and immediately exit if this fee is not present; the protocol allocated an allowance of 16 free execution steps to allow contracts to reject non-fee-paying transactions. this was the time when the ethereum protocol was entirely my own creation. from here on, however, new participants started to join the fold. by far the most prominent on the protocol side was gavin wood, who reached out to me in an about.me message in december 2013: jeffrey wilcke, lead developer of the go client (back then called "ethereal") also reached out and started coding around the same time, though his contributions were much more on the side of client development rather than protocol research. "hey jeremy, glad to see you're interested in ethereum..." gavin's initial contributions were two-fold. first, you might notice that the contract calling model in the initial design was an asynchronous one: although contract a could create an "internal transaction" to contract b ("internal transaction" is etherscan's lingo; initially they were just called "transactions" and then later "message calls" or "calls"), the internal transaction's execution would not start until the execution of the first transaction completely finished. this meant that transactions could not use internal transactions as a way of getting information from other contracts; the only way to do that was the extro opcode (kind of like an sload that you could use to read other contracts' storage), and this too was later removed with the support of gavin and others. when implementing my initial spec, gavin naturally implemented internal transactions synchronously without even realizing that the intent was different that is to say, in gavin's implementation, when a contract calls another contract, the internal transaction gets executed immediately, and once that execution finishes, the vm returns back to the contract that created the internal transaction and proceeds to the next opcode. this approach seemed to both of us to be superior, so we decided to make it part of the spec. second, a discussion between him and myself (during a walk in san francisco, so the exact details will be forever lost to the winds of history and possibly a copy or two in the deep archives of the nsa) led to a re-factoring of the transaction fee model, moving away from the "contract pays" approach to a "sender pays" approach, and also switching to the "gas" architecture. instead of each individual transaction step immediately taking away a bit of ether, the transaction sender pays for and is allocated some "gas" (roughly, a counter of computational steps), and computational steps drew from this allowance of gas. if a transaction runs out of gas, the gas would still be forfeit, but the entire execution would be reverted; this seemed like the safest thing to do, as it removed an entire class of "partial execution" attacks that contracts previously had to worry about. when a transaction execution finishes, the fee for any unused gas is refunded. gavin can also be largely credited for the subtle change in vision from viewing ethereum as a platform for building programmable money, with blockchain-based contracts that can hold digital assets and transfer them according to pre-set rules, to a general-purpose computing platform. this started with subtle changes in emphasis and terminology, and later this influence became stronger with the increasing emphasis on the "web 3" ensemble, which saw ethereum as being one piece of a suite of decentralized technologies, the other two being whisper and swarm. there were also changes made around the start of 2014 that were suggested by others. we ended up moving back to a stack-based architecture after the idea was suggested by andrew miller and others. charles hoskinson suggested the switch from bitcoin's sha256 to the newer sha3 (or, more accurately, keccak256). although there was some controversy for a while, discussions with gavin, andrew and others led to establishing that the size of values on the stack should be limited to 32 bytes; the other alternative being considered, unlimited-size integers, had the problem that it was too difficult to figure out how much gas to charge for additions, multiplications and other operations. the initial mining algorithm that we had in mind, back in january 2014, was a contraption called dagger: https://github.com/ethereum/wiki/blob/master/dagger.md dagger was named after the "directed acyclic graph" (dag), the mathematical structure that is used in the algorithm. the idea is that every n blocks, a new dag would be pseudorandomly generated from a seed, and the bottom layer of the dag would be a collection of nodes that takes several gigabytes to store. however, generating any individual value in the dag would require calculating only a few thousand entries. a "dagger computation" involved getting some number of values in random positions in this bottom-level dataset and hashing them together. this meant that there was a fast way to make a dagger calculation already having the data in memory, and a slow, but not memory intensive way regenerating each value from the dag that you need to get from scratch. the intention of this algorithm was to have the same "memory-hardness" properties as algorithms that were popular at the time, like scrypt, but still be light-client friendly. miners would use the fast way, and so their mining would be constrained by memory bandwidth (the theory is that consumer-grade ram is already very heavily optimized, and so it would be hard to further optimize it with asics), but light clients could use the memory-free but slower version for verification. the fast way might take a few microseconds and the slow but memory-free way a few milliseconds, so it would still be very viable for light clients. from here, the algorithm would change several times over the course of ethereum development. the next idea that we went through is "adaptive proof of work"; here, the proof of work would involve executing randomly selected ethereum contracts, and there is a clever reason why this is expected to be asic-resistant: if an asic was developed, competing miners would have the incentive to create and publish many contracts that that asic was not good at executing. there is no such thing as an asic for general computation, the story goes, as that is just a cpu, so we could instead use this kind of adversarial incentive mechanism to make a proof of work that essentially was executing general computation. this fell apart for one simple reason: long-range attacks. an attacker could start a chain from block 1, fill it up with only simple contracts that they can create specialized hardware for, and rapidly overtake the main chain. so... back to the drawing board. the next algorithm was something called random circuit, described in this google doc here, proposed by myself and vlad zamfir, and analyzed by matthew wampler-doty and others. the idea here was also to simulate general-purpose computation inside a mining algorithm, this time by executing randomly generated circuits. there's no hard proof that something based on these principles could not work, but the computer hardware experts that we reached out to in 2014 tended to be fairly pessimistic on it. matthew wampler-doty himself suggested a proof of work based on sat solving, but this too was ultimately rejected. finally, we came full circle with an algorithm called "dagger hashimoto". "dashimoto", as it was sometimes called in short, borrowed many ideas from hashimoto, a proof of work algorithm by thaddeus dryja that pioneered the notion of "i/o bound proof of work", where the dominant limiting factor in mining speed was not hashes per second, but rather megabytes per second of ram access. however, it combined this with dagger's notion of light-client-friendly dag-generated datasets. after many rounds of tweaking by myself, matthew, tim and others, the ideas finally converged into the algorithm we now call ethash. by the summer of 2014, the protocol had considerably stabilized, with the major exception of the proof of work algorithm which would not reach the ethash phase until around the beginning of 2015, and a semi-formal specification existed in the form of gavin's yellow paper. in august 2014, i developed and introduced the uncle mechanism, which allows ethereum's blockchain to have a shorter block time and higher capacity while mitigating centralization risks. this was introduced as part of poc6. discussions with the bitshares team led us to consider adding heaps as a first-class data structure, though we ended up not doing this due to lack of time, and later security audits and dos attacks will show that it is actually much harder than we had thought at the time to do this safely. in september, gavin and i planned out the next two major changes to the protocol design. first, alongside the state tree and transaction tree, every block would also contain a "receipt tree". the receipt tree would include hashes of the logs created by a transaction, along with intermediate state roots. logs would allow transactions to create "outputs" that are saved in the blockchain, and are accessible to light clients, but that are not accessible to future state calculations. this could be used to allow decentralized applications to easily query for events, such as token transfers, purchases, exchange orders being created and filled, auctions being started, and so forth. there were other ideas that were considered, like making a merkle tree out of the entire execution trace of a transaction to allow anything to be proven; logs were chosen because they were a compromise between simplicity and completeness. the second was the idea of "precompiles", solving the problem of allowing complex cryptographic computations to be usable in the evm without having to deal with evm overhead. we had also gone through many more ambitious ideas about "native contracts", where if miners have an optimized implementation of some contracts they could "vote" the gasprice of those contracts down, so contracts that most miners could execute much more quickly would naturally have a lower gas price; however, all of these ideas were rejected because we could not come up with a cryptoeconomically safe way to implement such a thing. an attacker could always create a contract which executes some trapdoored cryptographic operation, distribute the trapdoor to themselves and their friends to allow them to execute this contract much faster, then vote the gasprice down and use this to dos the network. instead we opted for the much less ambitious approach of having a smaller number of precompiles that are simply specified in the protocol, for common operations such as hashes and signature schemes. gavin was also a key initial voice in developing the idea of "protocol abstraction" moving as many parts of the protocol such as ether balances, transaction signing algorithms, nonces, etc into the protocol itself as contracts, with a theoretical final goal of reaching a situation where the entire ethereum protocol could be described as making a function call into a virtual machine that has some pre-initialized state. there was not enough time for these ideas to get into the initial frontier release, but the principles are expected to start slowly getting integrated through some of the constantinople changes, the casper contract and the sharding specification. this was all implemented in poc7; after poc7, the protocol did not really change much, with the exception of minor, though in some cases important, details that would come out through security audits... in early 2015, came the pre-launch security audits organized by jutta steiner and others, which included both software code audits and academic audits. the software audits were primarily on the c++ and go implementations, which were led by gavin wood and jeffrey wilcke, respectively, though there was also a smaller audit on my pyethereum implementation. of the two academic audits, one was performed by ittay eyal (of "selfish mining" fame), and the other by andrew miller and others from least authority. the eyal audit led to a minor protocol change: the total difficulty of a chain would not include uncles. the least authority audit was more focused on smart contract and gas economics, as well as the patricia tree. this audit led to several protocol changes. one small one is the use of sha3(addr) and sha3(key) as trie keys instead of the address and key directly; this would make it harder to perform a worst-case attack on the trie. and a warning that was perhaps a bit too far ahead of its time... another significant thing that we discussed was the gas limit voting mechanism. at the time, we were already concerned by perceived lack of progress in the bitcoin block size debate, and wanted to have a more flexible design in ethereum that could adjust over time as needed. but the challenge is: what is the optimal limit? my initial thought had been to make a dynamic limit, targeting \(1.5 \cdot\) the long-term exponential moving average of the actual gas usage, so that in the long run on average blocks would be \(\frac{2}{3}\) full. however, andrew showed that this was exploitable in some ways specifically, miners who wanted to raise the limit would simply include transactions in their own blocks that consume a very large amount of gas, but take very little time to process, and thereby always create full blocks at no cost to themselves. the security model was thus, at least in the upward direction, equivalent to simply having miners vote on the gas limit. we did not manage to come up with a gas limit strategy that was less likely to break, and so andrew's recommended solution was to simply have miners vote on the gas limit explicitly, and have the default strategy for voting be the \(1.5\cdot\) ema rule. the reasoning was that we were still very far from knowing the right approach for setting maximum gas limits, and the risk of any specific approach failing seemed greater than the risk of miners abusing their voting power. hence, we might as well simply let miners vote on the gas limit, and accept the risk that the limit will go too high or too low, in exchange for the benefit of flexibility, and the ability for miners to work together to very quickly adjust the limit upwards or downwards as needed. after a mini-hackathon between gavin, jeff and myself, poc9 was launched in march, and was intended to be the final proof of concept release. a testnet, olympic, ran for four months, using the protocol that was intended to be used in the livenet, and ethereum's long-term plan was established. vinay gupta wrote a blog post, "the ethereum launch process", that described the four expected stages of ethereum livenet development, and gave them their current names: frontier, homestead, metropolis and serenity. olympic ran for four months. in the first two months, many bugs were found in the various implementations, consensus failures happened, among other issues, but around june the network noticeably stabilized. in july a decision was made to make a code-freeze, followed by a release, and on july 30 the release took place. dark mode toggle governance, part 2: plutocracy is still bad 2018 mar 28 see all posts coin holder voting, both for governance of technical features, and for more extensive use cases like deciding who runs validator nodes and who receives money from development bounty funds, is unfortunately continuing to be popular, and so it seems worthwhile for me to write another post explaining why i (and vlad zamfir and others) do not consider it wise for ethereum (or really, any base-layer blockchain) to start adopting these kinds of mechanisms in a tightly coupled form in any significant way. i wrote about the issues with tightly coupled voting in a blog post last year, that focused on theoretical issues as well as focusing on some practical issues experienced by voting systems over the previous two years. now, the latest scandal in dpos land seems to be substantially worse. because the delegate rewards in eos are now so high (5% annual inflation, about $400m per year), the competition on who gets to run nodes has essentially become yet another frontier of us-china geopolitical economic warfare. and that's not my own interpretation; i quote from this article (original in chinese): eos supernode voting: multibillion-dollar profits leading to crypto community inter-country warfare looking at community recognition, chinese nodes feel much less represented in the community than us and korea. since the eos.io official twitter account was founded, there has never been any interaction with the mainland chinese eos community. for a listing of the eos officially promoted events and interactions with communities see the picture below. with no support from the developer community, facing competition from korea, the chinese eos supernodes have invented a new strategy: buying votes. the article then continues to describe further strategies, like forming "alliances" that all vote (or buy votes) for each other. of course, it does not matter at all who the specific actors are that are buying votes or forming cartels; this time it's some chinese pools, last time it was "members located in the usa, russia, india, germany, canada, italy, portugal and many other countries from around the globe", next time it could be totally anonymous, or run out of a smartphone snuck into trendon shavers's prison cell. what matters is that blockchains and cryptocurrency, originally founded in a vision of using technology to escape from the failures of human politics, have essentially all but replicated it. crypto is a reflection of the world at large. the eos new york community's response seems to be that they have issued a strongly worded letter to the world stating that buying votes will be against the constitution. hmm, what other major political entity has made accepting bribes a violation of the constitution? and how has that been going for them lately? the second part of this article will involve me, an armchair economist, hopefully convincing you, the reader, that yes, bribery is, in fact, bad. there are actually people who dispute this claim; the usual argument has something to do with market efficiency, as in "isn't this good, because it means that the nodes that win will be the nodes that can be the cheapest, taking the least money for themselves and their expenses and giving the rest back to the community?" the answer is, kinda yes, but in a way that's centralizing and vulnerable to rent-seeking cartels and explicitly contradicts many of the explicit promises made by most dpos proponents along the way. let us create a toy economic model as follows. there are a number of people all of which are running to be delegates. the delegate slot gives a reward of $100 per period, and candidates promise to share some portion of that as a bribe, equally split among all of their voters. the actual \(n\) delegates (eg. \(n = 35\)) in any period are the \(n\) delegates that received the most votes; that is, during every period a threshold of votes emerges where if you get more votes than that threshold you are a delegate, if you get less you are not, and the threshold is set so that \(n\) delegates are above the threshold. we expect that voters vote for the candidate that gives them the highest expected bribe. suppose that all candidates start off by sharing 1%; that is, equally splitting $1 among all of their voters. then, if some candidate becomes a delegate with \(k\) voters, each voter gets a payment of \(\frac{1}{k}\). the candidate that it's most profitable to vote for is a candidate that's expected to be in the top \(n\), but is expected to earn the fewest votes within that set. thus, we expect votes to be fairly evenly split among 35 delegates. now, some candidates will want to secure their position by sharing more; by sharing 2%, you are likely to get twice as many votes as those that share 1%, as that's the equilibrium point where voting for you has the same payout as voting for anyone else. the extra guarantee of being elected that this gives is definitely worth losing an additional 1% of your revenue when you do get elected. we can expect delegates to bid up their bribes and eventually share something close to 100% of their revenue. so the outcome seems to be that the delegate payouts are largely simply returned to voters, making the delegate payout mechanism close to meaningless. but it gets worse. at this point, there's an incentive for delegates to form alliances (aka political parties, aka cartels) to coordinate their share percentages; this reduces losses to the cartel from chaotic competition that accidentally leads to some delegates not getting enough votes. once a cartel is in place, it can start bringing its share percentages down, as dislodging it is a hard coordination problem: if a cartel offers 80%, then a new entrant offers 90%, then to a voter, seeking a share of that extra 10% is not worth the risk of either (i) voting for someone who gets insufficient votes and does not pay rewards, or (ii) voting for someone who gets too many votes and so pays out a reward that's excessively diluted. sidenote: bitshares dpos used approval voting, where you can vote for as many candidates as you want; it should be pretty obvious that with even slight bribery, the equilibrium there is that everyone just votes for everyone. furthermore, even if cartel mechanics don't come into play, there is a further issue. this equilibrium of coin holders voting for whoever gives them the most bribes, or a cartel that has become an entrenched rent seeker, contradicts explicit promises made by dpos proponents. quoting "explain delegated proof of stake like i'm 5": if a witness starts acting like an asshole, or stops doing a quality job securing the network, people in the community can remove their votes, essentially firing the bad actor. voting is always ongoing. from "eos: an introduction": by custom, we suggest that the bulk of the value be returned to the community for the common good software improvements, dispute resolution, and the like can be entertained. in the spirit of "eating our own dogfood," the design envisages that the community votes on a set of open entry contracts that act like "foundations" for the benefit of the community. known as community benefit contracts, the mechanism highlights the importance of dpos as enabling direct on-chain governance by the community (below). the flaw in all of this, of course, is that the average voter has only a very small chance of impacting which delegates get selected, and so they only have a very small incentive to vote based on any of these high-minded and lofty goals; rather, their incentive is to vote for whoever offers the highest and most reliable bribe. attacking is easy. if a cartel equilibrium does not form, then an attacker can simply offer a share percentage slightly higher than 100% (perhaps using fee sharing or some kind of "starter promotion" as justification), capture the majority of delegate positions, and then start an attack. if they get removed from the delegate position via a hard fork, they can simply restart the attack again with a different identity. the above is not intended purely as a criticism of dpos consensus or its use in any specific blockchain. rather, the critique reaches much further. there has been a large number of projects recently that extol the virtues of extensive on-chain governance, where on-chain coin holder voting can be used not just to vote on protocol features, but also to control a bounty fund. quoting a blog post from last year: anyone can submit a change to the governance structure in the form of a code update. an on-chain vote occurs, and if passed, the update makes its way on to a test network. after a period of time on the test network, a confirmation vote occurs, at which point the change goes live on the main network. they call this concept a "self-amending ledger". such a system is interesting because it shifts power towards users and away from the more centralized group of developers and miners. on the developer side, anyone can submit a change, and most importantly, everyone has an economic incentive to do it. contributions are rewarded by the community with newly minted tokens through inflation funding. this shifts from the current bitcoin and ethereum dynamics where a new developer has little incentive to evolve the protocol, thus power tends to concentrate amongst the existing developers, to one where everyone has equal earning power. in practice, of course, what this can easily lead to is funds that offer kickbacks to users who vote for them, leading to the exact scenario that we saw above with dpos delegates. in the best case, the funds will simply be returned to voters, giving coin holders an interest rate that cancels out the inflation, and in the worst case, some portion of the inflation will get captured as economic rent by a cartel. note also that the above is not a criticism of all on-chain voting; it does not rule out systems like futarchy. however, futarchy is untested, but coin voting is tested, and so far it seems to lead to a high risk of economic or political failure of some kind far too high a risk for a platform that seeks to be an economic base layer for development of decentralized applications and institutions. so what's the alternative? the answer is what we've been saying all along: cryptoeconomics. cryptoeconomics is fundamentally about the use of economic incentives together with cryptography to design and secure different kinds of systems and applications, including consensus protocols. the goal is simple: to be able to measure the security of a system (that is, the cost of breaking the system or causing it to violate certain guarantees) in dollars. traditionally, the security of systems often depends on social trust assumptions: the system works if 2 of 3 of alice, bob and charlie are honest, and we trust alice, bob and charlie to be honest because i know alice and she's a nice girl, bob registered with fincen and has a money transmitter license, and charlie has run a successful business for three years and wears a suit. social trust assumptions can work well in many contexts, but they are difficult to universalize; what is trusted in one country or one company or one political tribe may not be trusted in others. they are also difficult to quantify; how much money does it take to manipulate social media to favor some particular delegate in a vote? social trust assumptions seem secure and controllable, in the sense that "people" are in charge, but in reality they can be manipulated by economic incentives in all sorts of ways. cryptoeconomics is about trying to reduce social trust assumptions by creating systems where we introduce explicit economic incentives for good behavior and economic penalties for bad behavior, and making mathematical proofs of the form "in order for guarantee \(x\) to be violated, at least these people need to misbehave in this way, which means the minimum amount of penalties or foregone revenue that the participants suffer is \(y\)". casper is designed to accomplish precisely this objective in the context of proof of stake consensus. yes, this does mean that you can't create a "blockchain" by concentrating the consensus validation into 20 uber-powerful "supernodes" and you have to actually think to make a design that intelligently breaks through and navigates existing tradeoffs and achieves massive scalability in a still-decentralized network. but the reward is that you don't get a network that's constantly liable to breaking in half or becoming economically captured by unpredictable political forces. it has been brought to my attention that eos may be reducing its delegate rewards from 5% per year to 1% per year. needless to say, this doesn't really change the fundamental validity of any of the arguments; the only result of this would be 5x less rent extraction potential at the cost of a 5x reduction to the cost of attacking the system. some have asked: but how can it be wrong for dpos delegates to bribe voters, when it is perfectly legitimate for mining and stake pools to give 99% of their revenues back to their participants? the answer should be clear: in pow and pos, it's the protocol's role to determine the rewards that miners and validators get, based on the miners and validators' observed performance, and the fact that miners and validators that are pools pass along the rewards (and penalties!) to their participants gives the participants an incentive to participate in good pools. in dpos, the reward is constant, and it's the voters' role to vote for pools that have good performance, but with the key flaw that there is no mechanism to actually encourage voters to vote in that way instead of just voting for whoever gives them the most money without taking performance into account. penalties in dpos do not exist, and are certainly not passed on to voters, so voters have no "skin in the game" (penalties in casper pools, on the other hand, do get passed on to participants). dark mode toggle starks, part 3: into the weeds 2018 jul 21 see all posts special thanks to eli ben sasson for his kind assistance, as usual. special thanks to chih-cheng liang and justin drake for review, and to ben fisch for suggesting the reverse mimc technique for a vdf (paper here) trigger warning: math and lots of python as a followup to part 1 and part 2 of this series, this post will cover what it looks like to actually implement a stark, complete with an implementation in python. starks ("scalable transparent argument of knowledge" are a technique for creating a proof that \(f(x)=y\) where \(f\) may potentially take a very long time to calculate, but where the proof can be verified very quickly. a stark is "doubly scalable": for a computation with \(t\) steps, it takes roughly \(o(t \cdot \log{t})\) steps to produce a proof, which is likely optimal, and it takes ~\(o(\log^2{t})\) steps to verify, which for even moderately large values of \(t\) is much faster than the original computation. starks can also have a privacy-preserving "zero knowledge" property, though the use case we will apply them to here, making verifiable delay functions, does not require this property, so we do not need to worry about it. first, some disclaimers: this code has not been thoroughly audited; soundness in production use cases is not guaranteed this code is very suboptimal (it's written in python, what did you expect) starks "in real life" (ie. as implemented in eli and co's production implementations) tend to use binary fields and not prime fields for application-specific efficiency reasons; however, they do stress in their writings the prime field-based approach to starks described here is legitimate and can be used there is no "one true way" to do a stark. it's a broad category of cryptographic and mathematical constructs, with different setups optimal for different applications and constant ongoing research to reduce prover and verifier complexity and improve soundness. this article absolutely expects you to know how modular arithmetic and prime fields work, and be comfortable with the concepts of polynomials, interpolation and evaluation. if you don't, go back to part 2, and also this earlier post on quadratic arithmetic programs now, let's get to it. mimc here is the function we'll be doing a stark of: def mimc(inp, steps, round_constants): start_time = time.time() for i in range(steps-1): inp = (inp**3 + round_constants[i % len(round_constants)]) % modulus print("mimc computed in %.4f sec" % (time.time() start_time)) return inp we choose mimc (see paper) as the example because it is both (i) simple to understand and (ii) interesting enough to be useful in real life. the function can be viewed visually as follows: note: in many discussions of mimc, you will typically see xor used instead of +; this is because mimc is typically done over binary fields, where addition is xor; here we are doing it over prime fields. in our example, the round constants will be a relatively small list (eg. 64 items) that gets cycled through over and over again (that is, after k[64] it loops back to using k[1]). mimc with a very large number of rounds, as we're doing here, is useful as a verifiable delay function a function which is difficult to compute, and particularly non-parallelizable to compute, but relatively easy to verify. mimc by itself achieves this property to some extent because mimc can be computed "backward" (recovering the "input" from its corresponding "output"), but computing it backward takes about 100 times longer to compute than the forward direction (and neither direction can be significantly sped up by parallelization). so you can think of computing the function in the backward direction as being the act of "computing" the non-parallelizable proof of work, and computing the function in the forward direction as being the process of "verifying" it. \(x \rightarrow x^{(2p-1)/3}\) gives the inverse of \(x \rightarrow x^3\); this is true because of fermat's little theorem, a theorem that despite its supposed littleness is arguably much more important to mathematics than fermat's more famous "last theorem". what we will try to achieve here is to make verification much more efficient by using a stark instead of the verifier having to run mimc in the forward direction themselves, the prover, after completing the computation in the "backward direction", would compute a stark of the computation in the "forward direction", and the verifier would simply verify the stark. the hope is that the overhead of computing a stark can be less than the difference in speed running mimc forwards relative to backwards, so a prover's time would still be dominated by the initial "backward" computation, and not the (highly parallelizable) stark computation. verification of a stark can be relatively fast (in our python implementation, ~0.05-0.3 seconds), no matter how long the original computation is. all calculations are done modulo \(2^{256} 351 \cdot 2^{32} + 1\); we are using this prime field modulus because it is the largest prime below \(2^{256}\) whose multiplicative group contains an order \(2^{32}\) subgroup (that is, there's a number \(g\) such that successive powers of \(g\) modulo this prime loop around back to \(1\) after exactly \(2^{32}\) cycles), and which is of the form \(6k+5\). the first property is necessary to make sure that our efficient versions of the fft and fri algorithms can work, and the second ensures that mimc actually can be computed "backwards" (see the use of \(x \rightarrow x^{(2p-1)/3}\) above). prime field operations we start off by building a convenience class that does prime field operations, as well as operations with polynomials over prime fields. the code is here. first some trivial bits: class primefield(): def __init__(self, modulus): # quick primality test assert pow(2, modulus, modulus) == 2 self.modulus = modulus def add(self, x, y): return (x+y) % self.modulus def sub(self, x, y): return (x-y) % self.modulus def mul(self, x, y): return (x*y) % self.modulus and the extended euclidean algorithm for computing modular inverses (the equivalent of computing \(\frac{1}{x}\) in a prime field): # modular inverse using the extended euclidean algorithm def inv(self, a): if a == 0: return 0 lm, hm = 1, 0 low, high = a % self.modulus, self.modulus while low > 1: r = high//low nm, new = hm-lm*r, high-low*r lm, low, hm, high = nm, new, lm, low return lm % self.modulus the above algorithm is relatively expensive; fortunately, for the special case where we need to do many modular inverses, there's a simple mathematical trick that allows us to compute many inverses, called montgomery batch inversion: using montgomery batch inversion to compute modular inverses. inputs purple, outputs green, multiplication gates black; the red square is the only modular inversion. the code below implements this algorithm, with some slightly ugly special case logic so that if there are zeroes in the set of what we are inverting, it sets their inverse to 0 and moves along. def multi_inv(self, values): partials = [1] for i in range(len(values)): partials.append(self.mul(partials[-1], values[i] or 1)) inv = self.inv(partials[-1]) outputs = [0] * len(values) for i in range(len(values), 0, -1): outputs[i-1] = self.mul(partials[i-1], inv) if values[i-1] else 0 inv = self.mul(inv, values[i-1] or 1) return outputs this batch inverse algorithm will prove important later on, when we start dealing with dividing sets of evaluations of polynomials. now we move on to some polynomial operations. we treat a polynomial as an array, where element \(i\) is the \(i\)th degree term (eg. \(x^{3} + 2x + 1\) becomes [1, 2, 0, 1]). here's the operation of evaluating a polynomial at one point: # evaluate a polynomial at a point def eval_poly_at(self, p, x): y = 0 power_of_x = 1 for i, p_coeff in enumerate(p): y += power_of_x * p_coeff power_of_x = (power_of_x * x) % self.modulus return y % self.modulus challenge what is the output of f.eval_poly_at([4, 5, 6], 2) if the modulus is 31? mouseover below for answer \(6 \cdot 2^{2} + 5 \cdot 2 + 4 = 38, 38 \bmod 31 = 7\). there is also code for adding, subtracting, multiplying and dividing polynomials; this is textbook long addition/subtraction/multiplication/division. the one non-trivial thing is lagrange interpolation, which takes as input a set of x and y coordinates, and returns the minimal polynomial that passes through all of those points (you can think of it as being the inverse of polynomial evaluation): # build a polynomial that returns 0 at all specified xs def zpoly(self, xs): root = [1] for x in xs: root.insert(0, 0) for j in range(len(root)-1): root[j] -= root[j+1] * x return [x % self.modulus for x in root] def lagrange_interp(self, xs, ys): # generate master numerator polynomial, eg. (x x1) * (x x2) * ... * (x xn) root = self.zpoly(xs) # generate per-value numerator polynomials, eg. for x=x2, # (x x1) * (x x3) * ... * (x xn), by dividing the master # polynomial back by each x coordinate nums = [self.div_polys(root, [-x, 1]) for x in xs] # generate denominators by evaluating numerator polys at each x denoms = [self.eval_poly_at(nums[i], xs[i]) for i in range(len(xs))] invdenoms = self.multi_inv(denoms) # generate output polynomial, which is the sum of the per-value numerator # polynomials rescaled to have the right y values b = [0 for y in ys] for i in range(len(xs)): yslice = self.mul(ys[i], invdenoms[i]) for j in range(len(ys)): if nums[i][j] and ys[i]: b[j] += nums[i][j] * yslice return [x % self.modulus for x in b] see the "m of n" section of this article for a description of the math. note that we also have special-case methods lagrange_interp_4 and lagrange_interp_2 to speed up the very frequent operations of lagrange interpolation of degree \(< 2\) and degree \(< 4\) polynomials. fast fourier transforms if you read the above algorithms carefully, you might notice that lagrange interpolation and multi-point evaluation (that is, evaluating a degree \(< n\) polynomial at \(n\) points) both take quadratic time to execute, so for example doing a lagrange interpolation of one thousand points takes a few million steps to execute, and a lagrange interpolation of one million points takes a few trillion. this is an unacceptably high level of inefficiency, so we will use a more efficient algorithm, the fast fourier transform. the fft only takes \(o(n \cdot log(n))\) time (ie. ~10,000 steps for 1,000 points, ~20 million steps for 1 million points), though it is more restricted in scope; the x coordinates must be a complete set of roots of unity of some order \(n = 2^{k}\). that is, if there are \(n\) points, the x coordinates must be successive powers \(1, p, p^{2}, p^{3}\)... of some \(p\) where \(p^{n} = 1\). the algorithm can, surprisingly enough, be used for multi-point evaluation or interpolation, with one small parameter tweak. challenge find a 16th root of unity mod 337 that is not an 8th root of unity. mouseover below for answer 59, 146, 30, 297, 278, 191, 307, 40 you could have gotten this list by doing something like [print(x) for x in range(337) if pow(x, 16, 337) == 1 and pow(x, 8, 337) != 1], though there is a smarter way that works for much larger moduluses: first, identify a single primitive root mod 337 (that is, not a perfect square), by looking for a value x such that pow(x, 336 // 2, 337) != 1 (these are easy to find; one answer is 5), and then taking the (336 / 16)'th power of it. here's the algorithm (in a slightly simplified form; see code here for something slightly more optimized): def fft(vals, modulus, root_of_unity): if len(vals) == 1: return vals l = fft(vals[::2], modulus, pow(root_of_unity, 2, modulus)) r = fft(vals[1::2], modulus, pow(root_of_unity, 2, modulus)) o = [0 for i in vals] for i, (x, y) in enumerate(zip(l, r)): y_times_root = y*pow(root_of_unity, i, modulus) o[i] = (x+y_times_root) % modulus o[i+len(l)] = (x-y_times_root) % modulus return o def inv_fft(vals, modulus, root_of_unity): f = primefield(modulus) # inverse fft invlen = f.inv(len(vals)) return [(x*invlen) % modulus for x in fft(vals, modulus, f.inv(root_of_unity))] you can try running it on a few inputs yourself and check that it gives results that, when you use eval_poly_at on them, give you the answers you expect to get. for example: >>> fft.fft([3,1,4,1,5,9,2,6], 337, 85, inv=true) [46, 169, 29, 149, 126, 262, 140, 93] >>> f = poly_utils.primefield(337) >>> [f.eval_poly_at([46, 169, 29, 149, 126, 262, 140, 93], f.exp(85, i)) for i in range(8)] [3, 1, 4, 1, 5, 9, 2, 6] a fourier transform takes as input [x[0] .... x[n-1]], and its goal is to output x[0] + x[1] + ... + x[n-1] as the first element, x[0] + x[1] * 2 + ... + x[n-1] * w**(n-1) as the second element, etc etc; a fast fourier transform accomplishes this by splitting the data in half, doing an fft on both halves, and then gluing the result back together. a diagram of how information flows through the fft computation. notice how the fft consists of a "gluing" step followed by two copies of the fft on two halves of the data, and so on recursively until you're down to one element. i recommend this for more intuition on how or why the fft works and polynomial math in general, and this thread for some more specifics on dft vs fft, though be warned that most literature on fourier transforms talks about fourier transforms over real and complex numbers, not prime fields. if you find this too hard and don't want to understand it, just treat it as weird spooky voodoo that just works because you ran the code a few times and verified that it works, and you'll be fine too. thank goodness it's fri-day (that's "fast reed-solomon interactive oracle proofs of proximity") reminder: now may be a good time to review and re-read part 2 now, we'll get into the code for making a low-degree proof. to review, a low-degree proof is a (probabilistic) proof that at least some high percentage (eg. 80%) of a given set of values represent the evaluations of some specific polynomial whose degree is much lower than the number of values given. intuitively, just think of it as a proof that "some merkle root that we claim represents a polynomial actually does represent a polynomial, possibly with a few errors". as input, we have: a set of values that we claim are the evaluation of a low-degree polynomial a root of unity; the x coordinates at which the polynomial is evaluated are successive powers of this root of unity a value \(n\) such that we are proving the degree of the polynomial is strictly less than \(n\) the modulus our approach is a recursive one, with two cases. first, if the degree is low enough, we just provide the entire list of values as a proof; this is the "base case". verification of the base case is trivial: do an fft or lagrange interpolation or whatever else to interpolate the polynomial representing those values, and verify that its degree is \(< n\). otherwise, if the degree is higher than some set minimum, we do the vertical-and-diagonal trick described at the bottom of part 2. we start off by putting the values into a merkle tree and using the merkle root to select a pseudo-random x coordinate (special_x). we then calculate the "column": # calculate the set of x coordinates xs = get_power_cycle(root_of_unity, modulus) column = [] for i in range(len(xs)//4): x_poly = f.lagrange_interp_4( [xs[i+len(xs)*j//4] for j in range(4)], [values[i+len(values)*j//4] for j in range(4)], ) column.append(f.eval_poly_at(x_poly, special_x)) this packs a lot into a few lines of code. the broad idea is to re-interpret the polynomial \(p(x)\) as a polynomial \(q(x, y)\), where \(p(x) = q(x, x^4)\). if \(p\) has degree \(< n\), then \(p'(y) = q(special\_x, y)\) will have degree \(< \frac{n}{4}\). since we don't want to take the effort to actually compute \(q\) in coefficient form (that would take a still-relatively-nasty-and-expensive fft!), we instead use another trick. for any given value of \(x^{4}\), there are 4 corresponding values of \(x\): \(x\), \(modulus x\), and \(x\) multiplied by the two modular square roots of \(-1\). so we already have four values of \(q(?, x^4)\), which we can use to interpolate the polynomial \(r(x) = q(x, x^4)\), and from there calculate \(r(special\_x) = q(special\_x, x^4) = p'(x^4)\). there are \(\frac{n}{4}\) possible values of \(x^{4}\), and this lets us easily calculate all of them. a diagram from part 2; it helps to keep this in mind when understanding what's going on here our proof consists of some number (eg. 40) of random queries from the list of values of \(x^{4}\) (using the merkle root of the column as a seed), and for each query we provide merkle branches of the five values of \(q(?, x^4)\): m2 = merkelize(column) # pseudo-randomly select y indices to sample # (m2[1] is the merkle root of the column) ys = get_pseudorandom_indices(m2[1], len(column), 40) # compute the merkle branches for the values in the polynomial and the column branches = [] for y in ys: branches.append([mk_branch(m2, y)] + [mk_branch(m, y + (len(xs) // 4) * j) for j in range(4)]) the verifier's job will be to verify that these five values actually do lie on the same degree \(< 4\) polynomial. from there, we recurse and do an fri on the column, verifying that the column actually does have degree \(< \frac{n}{4}\). that really is all there is to fri. as a challenge exercise, you could try creating low-degree proofs of polynomial evaluations that have errors in them, and see how many errors you can get away passing the verifier with (hint, you'll need to modify the prove_low_degree function; with the default prover, even one error will balloon up and cause verification to fail). the stark reminder: now may be a good time to review and re-read part 1 now, we get to the actual meat that puts all of these pieces together: def mk_mimc_proof(inp, steps, round_constants) (code here), which generates a proof of the execution result of running the mimc function with the given input for some number of steps. first, some asserts: assert steps <= 2**32 // extension_factor assert is_a_power_of_2(steps) and is_a_power_of_2(len(round_constants)) assert len(round_constants) < steps the extension factor is the extent to which we will be "stretching" the computational trace (the set of "intermediate values" of executing the mimc function). we need the step count multiplied by the extension factor to be at most \(2^{32}\), because we don't have roots of unity of order \(2^{k}\) for \(k > 32\). our first computation will be to generate the computational trace; that is, all of the intermediate values of the computation, from the input going all the way to the output. # generate the computational trace computational_trace = [inp] for i in range(steps-1): computational_trace.append((computational_trace[-1]**3 + round_constants[i % len(round_constants)]) % modulus) output = computational_trace[-1] we then convert the computation trace into a polynomial, "laying down" successive values in the trace on successive powers of a root of unity \(g\) where \(g^{steps}\) = 1, and we then evaluate the polynomial in a larger set, of successive powers of a root of unity \(g_2\) where \((g_2)^{steps \cdot 8} = 1\) (note that \((g_2)^{8} = g\)). computational_trace_polynomial = inv_fft(computational_trace, modulus, subroot) p_evaluations = fft(computational_trace_polynomial, modulus, root_of_unity) black: powers of \(g_1\). purple: powers of \(g_2\). orange: 1. you can look at successive roots of unity as being arranged in a circle in this way. we are "laying" the computational trace along powers of \(g_1\), and then extending it compute the values of the same polynomial at the intermediate values (ie. the powers of \(g_2\)). we can convert the round constants of mimc into a polynomial. because these round constants loop around very frequently (in our tests, every 64 steps), it turns out that they form a degree-64 polynomial, and we can fairly easily compute its expression, and its extension: skips2 = steps // len(round_constants) constants_mini_polynomial = fft(round_constants, modulus, f.exp(subroot, skips2), inv=true) constants_polynomial = [0 if i % skips2 else constants_mini_polynomial[i//skips2] for i in range(steps)] constants_mini_extension = fft(constants_mini_polynomial, modulus, f.exp(root_of_unity, skips2)) suppose there are 8192 steps of execution and 64 round constants. here is what we are doing: we are doing an fft to compute the round constants as a function of \((g_1)^{128}\). we then add zeroes in between the constants to make it a function of \(g_1\) itself. because \((g_1)^{128}\) loops around every 64 steps, we know this function of \(g_1\) will as well. we only compute 512 steps of the extension, because we know that the extension repeats after 512 steps as well. we now, as in the fibonacci example in part 1, calculate \(c(p(x))\), except this time it's \(c(p(x), p(g_1 \cdot x), k(x))\): # create the composed polynomial such that # c(p(x), p(g1*x), k(x)) = p(g1*x) p(x)**3 k(x) c_of_p_evaluations = [(p_evaluations[(i+extension_factor)%precision] f.exp(p_evaluations[i], 3) constants_mini_extension[i % len(constants_mini_extension)]) % modulus for i in range(precision)] print('computed c(p, k) polynomial') note that here we are no longer working with polynomials in coefficient form; we are working with the polynomials in terms of their evaluations at successive powers of the higher-order root of unity. c_of_p is intended to be \(q(x) = c(p(x), p(g_1 \cdot x), k(x)) = p(g_1 \cdot x) p(x)^3 k(x)\); the goal is that for every \(x\) that we are laying the computational trace along (except for the last step, as there's no step "after" the last step), the next value in the trace is equal to the previous value in the trace cubed, plus the round constant. unlike the fibonacci example in part 1, where if one computational step was at coordinate \(k\), the next step is at coordinate \(k+1\), here we are laying down the computational trace along successive powers of the lower-order root of unity \(g_1\), so if one computational step is located at \(x = (g_1)^i\), the "next" step is located at \((g_1)^{i+1}\) = \((g_1)^i \cdot g_1 = x \cdot g_1\). hence, for every power of the lower-order root of unity \(g_1\) (except the last), we want it to be the case that \(p(x\cdot g_1) = p(x)^3 + k(x)\), or \(p(x\cdot g_1) p(x)^3 k(x) = q(x) = 0\). thus, \(q(x)\) will be equal to zero at all successive powers of the lower-order root of unity \(g\) (except the last). there is an algebraic theorem that proves that if \(q(x)\) is equal to zero at all of these x coordinates, then it is a multiple of the minimal polynomial that is equal to zero at all of these x coordinates: \(z(x) = (x x_1) \cdot (x x_2) \cdot ... \cdot (x x_n)\). since proving that \(q(x)\) is equal to zero at every single coordinate we want to check is too hard (as verifying such a proof would take longer than just running the original computation!), instead we use an indirect approach to (probabilistically) prove that \(q(x)\) is a multiple of \(z(x)\). and how do we do that? by providing the quotient \(d(x) = \frac{q(x)}{z(x)}\) and using fri to prove that it's an actual polynomial and not a fraction, of course! we chose the particular arrangement of lower and higher order roots of unity (rather than, say, laying the computational trace along the first few powers of the higher order root of unity) because it turns out that computing \(z(x)\) (the polynomial that evaluates to zero at all points along the computational trace except the last), and dividing by \(z(x)\) is trivial there: the expression of \(z\) is a fraction of two terms. # compute d(x) = q(x) / z(x) # z(x) = (x^steps 1) / (x x_atlast_step) z_num_evaluations = [xs[(i * steps) % precision] 1 for i in range(precision)] z_num_inv = f.multi_inv(z_num_evaluations) z_den_evaluations = [xs[i] last_step_position for i in range(precision)] d_evaluations = [cp * zd * zni % modulus for cp, zd, zni in zip(c_of_p_evaluations, z_den_evaluations, z_num_inv)] print('computed d polynomial') notice that we compute the numerator and denominator of \(z\) directly in "evaluation form", and then use the batch modular inversion to turn dividing by \(z\) into a multiplication (\(\cdot z_d \cdot z_ni\)), and then pointwise multiply the evaluations of \(q(x)\) by these inverses of \(z(x)\). note that at the powers of the lower-order root of unity except the last (ie. along the portion of the low-degree extension that is part of the original computational trace), \(z(x) = 0\), so this computation involving its inverse will break. this is unfortunate, though we will plug the hole by simply modifying the random checks and fri algorithm to not sample at those points, so the fact that we calculated them wrong will never matter. because \(z(x)\) can be expressed so compactly, we get another benefit: the verifier can compute \(z(x)\) for any specific \(x\) extremely quickly, without needing any precomputation. it's okay for the prover to have to deal with polynomials whose size equals the number of steps, but we don't want to ask the verifier to do the same, as we want verification to be succinct (ie. ultra-fast, with proofs as small as possible). probabilistically checking \(d(x) \cdot z(x) = q(x)\) at a few randomly selected points allows us to verify the transition constraints that each computational step is a valid consequence of the previous step. but we also want to verify the boundary constraints that the input and the output of the computation is what the prover says they are. just asking the prover to provide evaluations of \(p(1)\), \(d(1)\), \(p(last\_step)\) and \(d(last\_step)\) (where \(last\_step\) (or \(g^{steps-1}\)) is the coordinate corresponding to the last step in the computation) is too fragile; there's no proof that those values are on the same polynomial as the rest of the data. so instead we use a similar kind of polynomial division trick: # compute interpolant of ((1, input), (x_atlast_step, output)) interpolant = f.lagrange_interp_2([1, last_step_position], [inp, output]) i_evaluations = [f.eval_poly_at(interpolant, x) for x in xs] zeropoly2 = f.mul_polys([-1, 1], [-last_step_position, 1]) inv_z2_evaluations = f.multi_inv([f.eval_poly_at(quotient, x) for x in xs]) # b = (p i) / z2 b_evaluations = [((p i) * invq) % modulus for p, i, invq in zip(p_evaluations, i_evaluations, inv_z2_evaluations)] print('computed b polynomial') the argument is as follows. the prover wants to prove \(p(1) = input\) and \(p(last\_step) = output\). if we take \(i(x)\) as the interpolant the line that crosses the two points \((1, input)\) and \((last\_step, output)\), then \(p(x) i(x)\) would be equal to zero at those two points. thus, it suffices to prove that \(p(x) i(x)\) is a multiple of \((x 1) \cdot (x last\_step)\), and we do that by... providing the quotient! purple: computational trace polynomial (p). green: interpolant (i) (notice how the interpolant is constructed to equal the input (which should be the first step of the computational trace) at x=1 and the output (which should be the last step of the computational trace) at \(x=g^{steps-1}\). red: \(p i\). yellow: the minimal polynomial that equals \(0\) at \(x=1\) and \(x=g^{steps-1}\) (that is, \(z_2\)). pink: \(\frac{p i}{z_2}\). challenge suppose you wanted to also prove that the value in the computational trace after the 703rd computational step is equal to 8018284612598740. how would you modify the above algorithm to do that? mouseover below for answer set \(i(x)\) to be the interpolant of \((1, input), (g^{703}, 8018284612598740), (last\_step, output)\), and make a proof by providing the quotient \(b(x) = \frac{p(x) i(x)}{(x 1) \cdot (x g^{703}) \cdot (x last\_step)}\) now, we commit to the merkle root of \(p\), \(d\) and \(b\) combined together. # compute their merkle roots mtree = merkelize([pval.to_bytes(32, 'big') + dval.to_bytes(32, 'big') + bval.to_bytes(32, 'big') for pval, dval, bval in zip(p_evaluations, d_evaluations, b_evaluations)]) print('computed hash root') now, we need to prove that \(p\), \(d\) and \(b\) are all actually polynomials, and of the right max-degree. but fri proofs are big and expensive, and we don't want to have three fri proofs. so instead, we compute a pseudorandom linear combination of \(p\), \(d\) and \(b\) (using the merkle root of \(p\), \(d\) and \(b\) as a seed), and do an fri proof on that: k1 = int.from_bytes(blake(mtree[1] + b'\x01'), 'big') k2 = int.from_bytes(blake(mtree[1] + b'\x02'), 'big') k3 = int.from_bytes(blake(mtree[1] + b'\x03'), 'big') k4 = int.from_bytes(blake(mtree[1] + b'\x04'), 'big') # compute the linear combination. we don't even bother calculating it # in coefficient form; we just compute the evaluations root_of_unity_to_the_steps = f.exp(root_of_unity, steps) powers = [1] for i in range(1, precision): powers.append(powers[-1] * root_of_unity_to_the_steps % modulus) l_evaluations = [(d_evaluations[i] + p_evaluations[i] * k1 + p_evaluations[i] * k2 * powers[i] + b_evaluations[i] * k3 + b_evaluations[i] * powers[i] * k4) % modulus for i in range(precision)] unless all three of the polynomials have the right low degree, it's almost impossible that a randomly selected linear combination of them will (you have to get extremely lucky for the terms to cancel), so this is sufficient. we want to prove that the degree of d is less than \(2 \cdot steps\), and that of \(p\) and \(b\) are less than \(steps\), so we actually make a random linear combination of \(p\), \(p \cdot x^{steps}\), \(b\), \(b^{steps}\) and \(d\), and check that the degree of this combination is less than \(2 \cdot steps\). now, we do some spot checks of all of the polynomials. we generate some random indices, and provide the merkle branches of the polynomial evaluated at those indices: # do some spot checks of the merkle tree at pseudo-random coordinates, excluding # multiples of `extension_factor` branches = [] samples = spot_check_security_factor positions = get_pseudorandom_indices(l_mtree[1], precision, samples, exclude_multiples_of=extension_factor) for pos in positions: branches.append(mk_branch(mtree, pos)) branches.append(mk_branch(mtree, (pos + skips) % precision)) branches.append(mk_branch(l_mtree, pos)) print('computed %d spot checks' % samples) the get_pseudorandom_indices function returns some random indices in the range [0...precision-1], and the exclude_multiples_of parameter tells it to not give values that are multiples of the given parameter (here, extension_factor). this ensures that we do not sample along the original computational trace, where we are likely to get wrong answers. the proof (~250-500 kilobytes altogether) consists of a set of merkle roots, the spot-checked branches, and a low-degree proof of the random linear combination: o = [mtree[1], l_mtree[1], branches, prove_low_degree(l_evaluations, root_of_unity, steps * 2, modulus, exclude_multiples_of=extension_factor)] the largest parts of the proof in practice are the merkle branches, and the fri proof, which consists of even more branches. and here's the "meat" of the verifier: for i, pos in enumerate(positions): x = f.exp(g2, pos) x_to_the_steps = f.exp(x, steps) mbranch1 = verify_branch(m_root, pos, branches[i*3]) mbranch2 = verify_branch(m_root, (pos+skips)%precision, branches[i*3+1]) l_of_x = verify_branch(l_root, pos, branches[i*3 + 2], output_as_int=true) p_of_x = int.from_bytes(mbranch1[:32], 'big') p_of_g1x = int.from_bytes(mbranch2[:32], 'big') d_of_x = int.from_bytes(mbranch1[32:64], 'big') b_of_x = int.from_bytes(mbranch1[64:], 'big') zvalue = f.div(f.exp(x, steps) 1, x last_step_position) k_of_x = f.eval_poly_at(constants_mini_polynomial, f.exp(x, skips2)) # check transition constraints q(x) = z(x) * d(x) assert (p_of_g1x p_of_x ** 3 k_of_x zvalue * d_of_x) % modulus == 0 # check boundary constraints b(x) * z2(x) + i(x) = p(x) interpolant = f.lagrange_interp_2([1, last_step_position], [inp, output]) zeropoly2 = f.mul_polys([-1, 1], [-last_step_position, 1]) assert (p_of_x b_of_x * f.eval_poly_at(zeropoly2, x) f.eval_poly_at(interpolant, x)) % modulus == 0 # check correctness of the linear combination assert (l_of_x d_of_x k1 * p_of_x k2 * p_of_x * x_to_the_steps k3 * b_of_x k4 * b_of_x * x_to_the_steps) % modulus == 0 at every one of the positions that the prover provides a merkle proof for, the verifier checks the merkle proof, and checks that \(c(p(x), p(g_1 \cdot x), k(x)) = z(x) \cdot d(x)\) and \(b(x) \cdot z_2(x) + i(x) = p(x)\) (reminder: for \(x\) that are not along the original computation trace, \(z(x)\) will not be zero, and so \(c(p(x), p(g_1 \cdot x), k(x))\) likely will not evaluate to zero). the verifier also checks that the linear combination is correct, and calls verify_low_degree_proof(l_root, root_of_unity, fri_proof, steps * 2, modulus, exclude_multiples_of=extension_factor) to verify the fri proof. and we're done! well, not really; soundness analysis to prove how many spot-checks for the cross-polynomial checking and for the fri are necessary is really tricky. but that's all there is to the code, at least if you don't care about making even crazier optimizations. when i run the code above, we get a stark proving "overhead" of about 300-400x (eg. a mimc computation that takes 0.2 seconds to calculate takes 60 second to prove), suggesting that with a 4-core machine computing the stark of the mimc computation in the forward direction could actually be faster than computing mimc in the backward direction. that said, these are both relatively inefficient implementations in python, and the proving to running time ratio for properly optimized implementations may be different. also, it's worth pointing out that the stark proving overhead for mimc is remarkably low, because mimc is almost perfectly "arithmetizable" it's mathematical form is very simple. for "average" computations, which contain less arithmetically clean operations (eg. checking if a number is greater or less than another number), the overhead is likely much higher, possibly around 10000-50000x. dark mode toggle [mirror] bir proof of stake tasarım felsefesi 2016 dec 29 see all posts bu, https://medium.com/@vitalikbuterin/a-proof-of-stake-design-philosophy-506585978d51 adresindeki yazının bir aynası(kopyası)dır. ethereum gibi sistemler (ve bitcoin, nxt, bitshares vb.), temel olarak yeni bir kriptoekonomik organizma sınıfıdır merkeziyetsiz, yargı alanı olmayan, tamamen siber uzayda var olan ve kriptografi, ekonomi ve toplumsal fikir birliği (konsensüs) kombinasyonuyla sürdürülen varlıklardır. bittorrent gibi görünebilirler, ancak bittorrent'un state (durum) kavramı yoktur son derece önemli olduğu ortaya çıkan bir ayrım. bazen merkeziyetsiz otonom şirketler olarak tanımlanır, ancak tam anlamıyla şirket de değillerdir microsoft'u hard forklayamazsınız. açık kaynak yazılım projelerine benzeyebilir, ancak tam olarak da öyle değillerdir bir blok zincirini forklayabilirsiniz, ancak openoffice'i forklamak kadar kolay değildir. bu kriptoekonomik ağlar birçok farklı aromayla gelir asic-based pow, gpu-based pow, naive pos, delegated pos, umarım yakında casper pos ve her biri kaçınılmaz olarak kendi temel felsefesiyle birlikte gelir. i̇yi bilinen bir örnek, proof of work'ün maksimalist vizyonudur, burada "doğru" blok zinciri, madencilerin yaratmak için en büyük miktarda ekonomik sermayeyi harcadığı zincir olarak tanımlanır. başlangıçta yalnızca bir protokol içi fork choice kuralı olan bu mekanizma, birçok durumda kutsal bir dogma olarak yüceltilmiştir hash algoritmasını değiştiren protokol hard fork'ları karşısında bile fikrin saf halini ciddi şekilde savunmaya çalışan birinin bir örneği için chris derose ile benim aramdaki bu twitter tartışmasına bakın. bitshares'in delegated proof of stake'i başka bir tutarlı felsefeyi sunuyor, burada her şey yine tek bir temel ilke üzerinden akıyor, ancak çok daha da basit bir şekilde ifade edilebilir: hissedarlar oy kullanır. bu felsefelerin her biri; nakamoto konsensüsü, sosyal konsensüs, hissedar oyları konsensüsü, kendi sonuçlarına götürür ve kendi terimleriyle bakıldığında oldukça anlamlı olan bir değerler sistemine yol açar ancak birbirleriyle karşılaştırıldıklarında kesinlikle eleştirilebilirler. casper konsensüsünün de felsefi bir temeli var, ancak bu şimdiye kadar açık bir şekilde makaleye alınmamıştır. ben, vlad, dominic, jae ve diğerleri, proof of stake protokollerinin neden var olduğu ve nasıl tasarlandığı konusunda kendi görüşlerimize sahibiz, ancak burada kişisel olarak nereden geldiğimi açıklamayı amaçlıyorum. gözlemlerimi ve ardından direkt sonuçları listeleyeceğim. kriptografi, 21. yüzyılda gerçekten özeldir çünkü kriptografi, düşmanca çatışmalarda büyük ölçüde savunanın lehine olmaya devam ettiği çok az alandan biridir. kaleleri yok etmek inşa etmekten çok daha kolaydır, adalar savunulabilir ancak yine de saldırılabilir, ancak ortalama bir kişinin ecc anahtarları devlet düzeyindeki aktörlere bile direnecek kadar güvenlidir. cypherpunk felsefesi, temel olarak, bireyin özerkliğini daha iyi koruyan bir dünya yaratmak için bu değerli asimetriden yararlanmakla ilgilidir ve kriptoekonomi bir dereceye kadar bunun bir uzantısıdır, ancak bu sefer sadece özel iletilerin bütünlüğü ve gizliliği değil, karmaşık koordinasyon ve işbirliği sistemlerinin güvenliği ve canlılığını koruması gerekir. kendilerini cypherpunk ruhunun ideolojik mirasçıları olarak gören sistemler, bu temel özelliği korumalı ve sistemi yok etmeleri veya bozmaları, sistemi kullanıp sürdürmekten çok daha pahalı olmalıdır. "cypherpunk ruhu" sadece idealizmle ilgili değildir; savunması saldırmaktan daha kolay olan sistemler yapmak da aslında sağlam mühendisliktir. orta ila uzun zaman ölçeklerinde, insanlar fikir birliğinde (konsensüs) oldukça iyidir. bir düşman sınırsız hash gücüne erişimi olsa ve herhangi bir büyük blokzincirine %51 saldırısı gerçekleştirerek tarihin son ayını geri alsa bile, topluluğu bu zincirin doğru olan olduğuna ikna etmek, ana zincirin hash gücünü aşmaktan çok daha zor. blok explorerları, topluluktaki her güvenilir üyeyi, new york times'ı, archive.org'u ve internetteki diğer birçok kaynağı yıkmaları gerekecektir; genel olarak, bilgi teknolojisinin yoğun olduğu 21. yüzyılda yeni saldırı zincirinin asıl zincir olduğunu dünyaya ikna etmek, abd'nin ay'a inişinin gerçekleşmediğini dünyaya ikna etmek kadar zordur. bu sosyal dikkate almalar, blokzinciri topluluğunun kabul edip etmemesine bakılmaksızın, herhangi bir blok zincirini uzun vadede koruyan şeydir (dikkate alın, bitcoin core sosyal katmanın önemini kabul ediyor). ancak tek başına sosyal konsensüs ile korunan bir blokzinciri, çok verimsiz ve yavaş olacaktır ve bitmeyen anlaşmazlıkların gerçekleşmesi daha kolay olacaktır (tüm zorluklara rağmen, böyle bir şey yaşandı); dolayısıyla ekonomik konsensüs, kısa vadede canlılık ve güvenlik özelliklerinin korunmasında son derece önemli bir rol oynuyor. proof of work güvenliği sadece blok ödüllerinden geliyor (dominic williams'ın deyimiyle, üç e'den ikisi eksik), madencilerin teşviki yalnızca gelecekteki blok ödüllerini kaybetme riskiyle geliyor. bu nedenle, proof of work mantıksal olarak büyük ödüllerle var olmasına teşvik edilen büyük güç üzerine dayanır. pow'da saldırılardan kurtulmak çok zordur: i̇lk kez gerçekleştiğinde pow'i değiştirmek için bir hard fork yapabilir ve saldırganın asic'lerini işe yaramaz hale getirebilirsiniz, ancak ikinci kez bu seçeneğiniz olmaz ve saldırgan tekrar tekrar saldırabilir. bu nedenle, madencilik ağı o kadar büyük olmalıdır ki saldırılar düşünülemez hale gelmelidir. x'ten daha küçük saldırganlar, ağın her gün sürekli olarak x harcamasıyla ortaya çıkmaktan (saldırmaktan) vazgeçirilir. ben bu mantığı reddediyorum çünkü (i) ağaçları öldürüyor ve (ii) cypherpunk ruhunu tanımayı başaramıyor saldırı maliyeti ve savunma maliyeti 1:1 oranında, bu yüzden herhangi bir savunma avantajı yok. proof of stake, güvenliği ödüllere dayanmak yerine cezalara dayandırarak bu simetriyi kırar. doğrulayıcılar (validator) para ("mevduat") riske eder, sermayelerini kitlemeleri ve node'larını sürdürmeleri ve private key'lerinin güvenliğini sağlamak için önlemler almaları karşılığında bir miktar ödül alır, ancak işlemleri geri alma maliyetinin büyük bir kısmı, bu süre zarfında aldıkları ödüllerden yüzlerce veya binlerce kat daha büyük olan cezalardan kaynaklanır. bu nedenle proof of stake'in tek cümlelik felsefesi, "güvenlik yanan enerjiden gelir" değil, "güvenlik, ekonomik kayba dayalı değer koymaktan gelir" şeklindedir. kötü niyetli node'lar, takasın x$ değerinde protokol içi cezalar ödemesini sağlamak için suç ortağı olmadıkça, tutarsız herhangi bir blok veya durum için eşit bir sonlandırma düzeyi elde etmenin mümkün olmadığını kanıtlayabilirseniz, belirli bir blok veya durum x$ güvenliğine sahiptir. teorik olarak, çoğunluktaki doğrulayıcıların(validator) gizli anlaşmaları bir pos zincirini devralabilir ve kötü niyetli davranmaya başlayabilir. ancak, (i) akıllıca bir protokol tasarımıyla, bu tür manipülasyonlar ile ek kar elde etme yetenekleri mümkün olduğunca sınırlanabilir ve daha da önemlisi (ii) eğer yeni doğrulayıcıların katılmasını engellemeye çalışırlar veya %51 saldırısı gerçekleştirirlerse, topluluk basitçe bir hard fork koordine ederek söz konusu kötü niyetli doğrulayıcıların mevduatlarını silme yetisine sahiptir. başarılı bir saldırı 50 milyon dolara mal olabilir, ancak sonuçlarının temizlenmesi süreci, 2016.11.25'teki geth/parity konsensüs hatasından çok daha külfetli olmayacaktır. i̇ki gün sonra, blokzinciri ve topluluk eski haline dönerken, saldırganlar 50 milyon dolar daha fakir ve topluluğun geri kalanı muhtemelen daha zengin çünkü saldırı, müteakip arz sıkışıklığı nedeniyle tokenin değerinin yükselmesine neden oldu. i̇şte size saldırı/savunma asimetrisi. yukarıdaki ifade, plansız hard fork'ların düzenli bir şekilde meydana geleceği anlamına gelmemelidir. i̇stenirse, proof of stake'teki bir tane 51% saldırısının maliyeti, proof of work'te kalıcı 51% saldırısı yapmakla aynı derecede yüksek olabilir ve saldırının yüksek maliyeti ve etkisizliği, pratikte neredeyse hiç denemeye kalkışılmamasını sağlar. ekonomi her şey değildir. bireysel aktörler, protokol dışı nedenlerle davranabilirler, hacklenebilirler, kaçırılabilirler veya sadece bir gün sarhoş olup blokzincirini mahvetmeye ve maliyetini umursamamaya karar verebilirler. ayrıca, olumlu yönden baktığımızda, bireylerin ahlaki sınırlamaları ve iletişim eksiklikleri, bir saldırının maliyetini nominal protokol tanımlı değer kaybı seviyesinden çok daha yüksek seviyelere çıkaracaktır. bu, güvenemeyeceğimiz bir avantaj, ancak aynı zamanda gereksiz yere görmezden gelmememiz gereken bir avantajdır. bu nedenle, en iyi protokoller, çeşitli modeller ve varsayımlar altında iyi çalışan protokollerdir koordine edilmiş seçimlerle ekonomik rasyonellik, bireysel seçimlerle ekonomik rasyonellik, basit hata toleransı, bizans hata toleransı (ideal olarak hem uyarlamalı hem de uyarlamasız saldırgan varyantları), ariely/kahneman'dan esinlenmiş davranışsal ekonomik modeller ("hepimiz biraz hile yaparız") ve mümkünse gerçekçi ve mantıklı düşünülebilen diğer herhangi bir model. her iki savunma katmanına da sahip olmak önemlidir: merkezi kartellerin anti-sosyal davranmasını engellemek için ekonomik teşvikler, en başta kartellerin oluşmasını engellemek için anti-merkeziyetçi teşvikler. olabildiğince hızlı çalışan konsensüs protokollerinin riskleri vardır ve eğer varsa çok dikkatli bir şekilde yaklaşılmalıdır, çünkü çok hızlı olma olasılığı bunu yapmaya yönelik teşviklere bağlıysa, bu kombinasyon ağ düzeyinde çok yüksek ve sistemik risk oluşturan bir merkezileşme seviyesini ödüllendirir (örneğin, tüm doğrulayıcıların aynı hosting sağlayıcısından çalışması). doğrulayıcıların bir mesajı ne kadar hızlı gönderdiğini çok fazla umursamayan konsensüs protokolleri, bunu kabul edilebilir uzun bir zaman aralığı içinde yaptıkları sürece (örneğin, 4-8 saniye gibi, çünkü ethereum'daki gecikme süresinin genellikle ~500ms-1s olduğunu gözlemledik) böyle endişeler taşımaz. mümkün olan bir orta nokta, çok hızlı çalışabilen ancak ethereum'un amca(uncle) mekanizmasına benzer mekanikler taşıyan, bir node'un ağ bağlantısının kolayca ulaşılabilecek bir noktanın ötesine geçmesi durumunda marjinal ödülün oldukça düşük olduğunu sağlayan, protokoller oluşturmaktır. buradan itibaren, elbette birçok detay ve detaylarda farklı yollar var, ancak yukarıdakiler en azından benim casper versiyonumun dayandığı temel prensipleridir. buradan itibaren, rekabet eden değerler arasında tercihleri tartışabiliriz. eth'e yıllık %1 oranında bir arz oranı verelim ve düzeltici bir hard fork için 50 milyon dolar maliyet alalım mı, yoksa yıllık %0 oranında bir arz oranı verelim ve düzeltici bir hard fork için 5 milyon dolar maliyet alalım mı? ekonomik model altında bir protokolün güvenliğini artırmak için hata tolere edilebilirlik modeli altında güvenliğini azaltma değiş tokuşunu ne zaman yaparız? tahmin edilebilir bir güvenlik seviyesine mi yoksa tahmin edilebilir bir arz seviyesine mi daha çok önem veriyoruz? bu soruların hepsi başka bir post'un konusu ve bu değerler arasındaki farklı değiş tokuşları uygulamanın çeşitli yolları, daha da fazla post'un konusu. ama onlara da geleceğiz :) dark mode toggle [mirror] quadratic arithmetic programs: from zero to hero 2016 dec 10 see all posts this is a mirror of the post at https://medium.com/@vitalikbuterin/quadratic-arithmetic-programs-from-zero-to-hero-f6d558cea649 there has been a lot of interest lately in the technology behind zk-snarks, and people are increasingly trying to demystify something that many have come to call "moon math" due to its perceived sheer indecipherable complexity. zk-snarks are indeed quite challenging to grasp, especially due to the sheer number of moving parts that need to come together for the whole thing to work, but if we break the technology down piece by piece then comprehending it becomes simpler. the purpose of this post is not to serve as a full introduction to zk-snarks; it assumes as background knowledge that (i) you know what zk-snarks are and what they do, and (ii) know enough math to be able to reason about things like polynomials (if the statement \(p(x) + q(x) = (p + q)(x)\) , where \(p\) and \(q\) are polynomials, seems natural and obvious to you, then you're at the right level). rather, the post digs deeper into the machinery behind the technology, and tries to explain as well as possible the first half of the pipeline, as drawn by zk-snark researcher eran tromer here: the steps here can be broken up into two halves. first, zk-snarks cannot be applied to any computational problem directly; rather, you have to convert the problem into the right "form" for the problem to operate on. the form is called a "quadratic arithmetic program" (qap), and transforming the code of a function into one of these is itself highly nontrivial. along with the process for converting the code of a function into a qap is another process that can be run alongside so that if you have an input to the code you can create a corresponding solution (sometimes called "witness" to the qap). after this, there is another fairly intricate process for creating the actual "zero knowledge proof" for this witness, and a separate process for verifying a proof that someone else passes along to you, but these are details that are out of scope for this post. the example that we will choose is a simple one: proving that you know the solution to a cubic equation: \(x^3 + x + 5 = 35\) (hint: the answer is \(3\)). this problem is simple enough that the resulting qap will not be so large as to be intimidating, but nontrivial enough that you can see all of the machinery come into play. let us write out our function as follows: def qeval(x): y = x**3 return x + y + 5 the simple special-purpose programming language that we are using here supports basic arithmetic (\(+\), \(-\), \(\cdot\), \(/\)), constant-power exponentiation (\(x^7\) but not \(x^y\)) and variable assignment, which is powerful enough that you can theoretically do any computation inside of it (as long as the number of computational steps is bounded; no loops allowed). note that modulo (%) and comparison operators (\(<\), \(>\), \(\leq\), \(\geq\)) are not supported, as there is no efficient way to do modulo or comparison directly in finite cyclic group arithmetic (be thankful for this; if there was a way to do either one, then elliptic curve cryptography would be broken faster than you can say "binary search" and "chinese remainder theorem"). you can extend the language to modulo and comparisons by providing bit decompositions (eg. \(13 = 2^3 + 2^2 + 1\)) as auxiliary inputs, proving correctness of those decompositions and doing the math in binary circuits; in finite field arithmetic, doing equality (==) checks is also doable and in fact a bit easier, but these are both details we won't get into right now. we can extend the language to support conditionals (eg. if \(x < 5: y = 7;\) else: \(y = 9\)) by converting them to an arithmetic form: \(y = 7 \cdot (x < 5) + 9 \cdot (x \geq 5)\) though note that both "paths" of the conditional would need to be executed, and if you have many nested conditionals then this can lead to a large amount of overhead. let us now go through this process step by step. if you want to do this yourself for any piece of code, i implemented a compiler here (for educational purposes only; not ready for making qaps for real-world zk-snarks quite yet!). flattening the first step is a "flattening" procedure, where we convert the original code, which may contain arbitrarily complex statements and expressions, into a sequence of statements that are of two forms: \(x = y\) (where \(y\) can be a variable or a number) and \(x = y\) \((op)\) \(z\) (where \(op\) can be \(+\), \(-\), \(\cdot\), \(/\) and \(y\) and \(z\) can be variables, numbers or themselves sub-expressions). you can think of each of these statements as being kind of like logic gates in a circuit. the result of the flattening process for the above code is as follows: sym_1 = x * x y = sym_1 * x sym_2 = y + x ~out = sym_2 + 5 if you read the original code and the code here, you can fairly easily see that the two are equivalent. gates to r1cs now, we convert this into something called a rank-1 constraint system (r1cs). an r1cs is a sequence of groups of three vectors (\(a\), \(b\), \(c\)), and the solution to an r1cs is a vector \(s\), where \(s\) must satisfy the equation \(s . a \cdot s . b s . c = 0\), where \(.\) represents the dot product in simpler terms, if we "zip together" \(a\) and \(s\), multiplying the two values in the same positions, and then take the sum of these products, then do the same to \(b\) and \(s\) and then \(c\) and \(s\), then the third result equals the product of the first two results. for example, this is a satisfied r1cs: but instead of having just one constraint, we are going to have many constraints: one for each logic gate. there is a standard way of converting a logic gate into a \((a, b, c)\) triple depending on what the operation is (\(+\), \(-\), \(\cdot\) or \(/\)) and whether the arguments are variables or numbers. the length of each vector is equal to the total number of variables in the system, including a dummy variable ~one at the first index representing the number \(1\), the input variables, a dummy variable ~out representing the output, and then all of the intermediate variables (\(sym_1\) and \(sym_2\) above); the vectors are generally going to be very sparse, only filling in the slots corresponding to the variables that are affected by some particular logic gate. first, we'll provide the variable mapping that we'll use: '~one', 'x', '~out', 'sym_1', 'y', 'sym_2' the solution vector will consist of assignments for all of these variables, in that order. now, we'll give the \((a, b, c)\) triple for the first gate: a = [0, 1, 0, 0, 0, 0] b = [0, 1, 0, 0, 0, 0] c = [0, 0, 0, 1, 0, 0] you can see that if the solution vector contains \(3\) in the second position, and \(9\) in the fourth position, then regardless of the rest of the contents of the solution vector, the dot product check will boil down to \(3 \cdot 3 = 9\), and so it will pass. if the solution vector has \(-3\) in the second position and \(9\) in the fourth position, the check will also pass; in fact, if the solution vector has \(7\) in the second position and \(49\) in the fourth position then that check will still pass — the purpose of this first check is to verify the consistency of the inputs and outputs of the first gate only. now, let's go on to the second gate: a = [0, 0, 0, 1, 0, 0] b = [0, 1, 0, 0, 0, 0] c = [0, 0, 0, 0, 1, 0] in a similar style to the first dot product check, here we're checking that \(sym_1 \cdot x = y\). now, the third gate: a = [0, 1, 0, 0, 1, 0] b = [1, 0, 0, 0, 0, 0] c = [0, 0, 0, 0, 0, 1] here, the pattern is somewhat different: it's multiplying the first element in the solution vector by the second element, then by the fifth element, adding the two results, and checking if the sum equals the sixth element. because the first element in the solution vector is always one, this is just an addition check, checking that the output equals the sum of the two inputs. finally, the fourth gate: a = [5, 0, 0, 0, 0, 1] b = [1, 0, 0, 0, 0, 0] c = [0, 0, 1, 0, 0, 0] here, we're evaluating the last check, ~out \(= sym_2 + 5\). the dot product check works by taking the sixth element in the solution vector, adding five times the first element (reminder: the first element is \(1\), so this effectively means adding \(5\)), and checking it against the third element, which is where we store the output variable. and there we have our r1cs with four constraints. the witness is simply the assignment to all the variables, including input, output and internal variables: [1, 3, 35, 9, 27, 30] you can compute this for yourself by simply "executing" the flattened code above, starting off with the input variable assignment \(x=3\), and putting in the values of all the intermediate variables and the output as you compute them. the complete r1cs put together is: a [0, 1, 0, 0, 0, 0] [0, 0, 0, 1, 0, 0] [0, 1, 0, 0, 1, 0] [5, 0, 0, 0, 0, 1] b [0, 1, 0, 0, 0, 0] [0, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] c [0, 0, 0, 1, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [0, 0, 1, 0, 0, 0] r1cs to qap the next step is taking this r1cs and converting it into qap form, which implements the exact same logic except using polynomials instead of dot products. we do this as follows. we go 3from four groups of three vectors of length six to six groups of three degree-3 polynomials, where evaluating the polynomials at each x coordinate represents one of the constraints. that is, if we evaluate the polynomials at \(x=1\), then we get our first set of vectors, if we evaluate the polynomials at \(x=2\), then we get our second set of vectors, and so on. we can make this transformation using something called a lagrange interpolation. the problem that a lagrange interpolation solves is this: if you have a set of points (ie. \((x, y)\) coordinate pairs), then doing a lagrange interpolation on those points gives you a polynomial that passes through all of those points. we do this by decomposing the problem: for each \(x\) coordinate, we create a polynomial that has the desired \(y\) coordinate at that \(x\) coordinate and a \(y\) coordinate of \(0\) at all the other \(x\) coordinates we are interested in, and then to get the final result we add all of the polynomials together. let's do an example. suppose that we want a polynomial that passes through \((1, 3), (2, 2)\) and \((3, 4)\). we start off by making a polynomial that passes through \((1, 3), (2, 0)\) and \((3, 0)\). as it turns out, making a polynomial that "sticks out" at \(x=1\) and is zero at the other points of interest is easy; we simply do: (x 2) * (x 3) which looks like this: now, we just need to "rescale" it so that the height at x=1 is right: (x 2) * (x 3) * 3 / ((1 2) * (1 3)) this gives us: 1.5 * x**2 7.5 * x + 9 we then do the same with the other two points, and get two other similar-looking polynomials, except that they "stick out" at \(x=2\) and \(x=3\) instead of \(x=1\). we add all three together and get: 1.5 * x**2 5.5 * x + 7 with exactly the coordinates that we want. the algorithm as described above takes \(o(n^3)\) time, as there are \(n\) points and each point requires \(o(n^2)\) time to multiply the polynomials together; with a little thinking, this can be reduced to \(o(n^2)\) time, and with a lot more thinking, using fast fourier transform algorithms and the like, it can be reduced even further — a crucial optimization when functions that get used in zk-snarks in practice often have many thousands of gates. now, let's use lagrange interpolation to transform our r1cs. what we are going to do is take the first value out of every \(a\) vector, use lagrange interpolation to make a polynomial out of that (where evaluating the polynomial at \(i\) gets you the first value of the \(i\)th \(a\) vector), repeat the process for the first value of every \(b\) and \(c\) vector, and then repeat that process for the second values, the third, values, and so on. for convenience i'll provide the answers right now: a polynomials [-5.0, 9.166, -5.0, 0.833] [8.0, -11.333, 5.0, -0.666] [0.0, 0.0, 0.0, 0.0] [-6.0, 9.5, -4.0, 0.5] [4.0, -7.0, 3.5, -0.5] [-1.0, 1.833, -1.0, 0.166] b polynomials [3.0, -5.166, 2.5, -0.333] [-2.0, 5.166, -2.5, 0.333] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] c polynomials [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [-1.0, 1.833, -1.0, 0.166] [4.0, -4.333, 1.5, -0.166] [-6.0, 9.5, -4.0, 0.5] [4.0, -7.0, 3.5, -0.5] coefficients are in ascending order, so the first polynomial above is actually \(0.833 \cdot x^3 — 5 \cdot x^2 + 9.166 \cdot x 5\). this set of polynomials (plus a z polynomial that i will explain later) makes up the parameters for this particular qap instance. note that all of the work up until this point needs to be done only once for every function that you are trying to use zk-snarks to verify; once the qap parameters are generated, they can be reused. let's try evaluating all of these polynomials at \(x=1\). evaluating a polynomial at \(x=1\) simply means adding up all the coefficients (as \(1^k = 1\) for all \(k\)), so it's not difficult. we get: a results at x=1 0 1 0 0 0 0 b results at x=1 0 1 0 0 0 0 c results at x=1 0 0 0 1 0 0 and lo and behold, what we have here is exactly the same as the set of three vectors for the first logic gate that we created above. checking the qap now what's the point of this crazy transformation? the answer is that instead of checking the constraints in the r1cs individually, we can now check all of the constraints at the same time by doing the dot product check on the polynomials. because in this case the dot product check is a series of additions and multiplications of polynomials, the result is itself going to be a polynomial. if the resulting polynomial, evaluated at every \(x\) coordinate that we used above to represent a logic gate, is equal to zero, then that means that all of the checks pass; if the resulting polynomial evaluated at at least one of the \(x\) coordinate representing a logic gate gives a nonzero value, then that means that the values going into and out of that logic gate are inconsistent (ie. the gate is \(y = x \cdot sym_1\) but the provided values might be \(x = 2,sym_1 = 2\) and \(y = 5\)). note that the resulting polynomial does not itself have to be zero, and in fact in most cases won't be; it could have any behavior at the points that don't correspond to any logic gates, as long as the result is zero at all the points that do correspond to some gate. to check correctness, we don't actually evaluate the polynomial \(t = a . s \cdot b . s c . s\) at every point corresponding to a gate; instead, we divide \(t\) by another polynomial, \(z\), and check that \(z\) evenly divides \(t\) that is, the division \(t / z\) leaves no remainder. \(z\) is defined as \((x 1) \cdot (x 2) \cdot (x 3) ...\) the simplest polynomial that is equal to zero at all points that correspond to logic gates. it is an elementary fact of algebra that any polynomial that is equal to zero at all of these points has to be a multiple of this minimal polynomial, and if a polynomial is a multiple of \(z\) then its evaluation at any of those points will be zero; this equivalence makes our job much easier. now, let's actually do the dot product check with the polynomials above. first, the intermediate polynomials: a . s = [43.0, -73.333, 38.5, -5.166] b . s = [-3.0, 10.333, -5.0, 0.666] c . s = [-41.0, 71.666, -24.5, 2.833] now, \(a . s \cdot b . s — c . s\): t = [-88.0, 592.666, -1063.777, 805.833, -294.777, 51.5, -3.444] now, the minimal polynomial \(z = (x 1) \cdot (x 2) \cdot (x 3) \cdot (x 4)\): z = [24, -50, 35, -10, 1] and if we divide the result above by \(z\), we get: h = t / z = [-3.666, 17.055, -3.444] with no remainder. and so we have the solution for the qap. if we try to falsify any of the variables in the r1cs solution that we are deriving this qap solution from — say, set the last one to \(31\) instead of \(30\), then we get a \(t\) polynomial that fails one of the checks (in that particular case, the result at \(x=3\) would equal \(-1\) instead of \(0\)), and furthermore \(t\) would not be a multiple of \(z\); rather, dividing \(t / z\) would give a remainder of \([-5.0, 8.833, -4.5, 0.666]\). note that the above is a simplification; "in the real world", the addition, multiplication, subtraction and division will happen not with regular numbers, but rather with finite field elements — a spooky kind of arithmetic which is self-consistent, so all the algebraic laws we know and love still hold true, but where all answers are elements of some finite-sized set, usually integers within the range from \(0\) to \(n-1\) for some \(n\). for example, if \(n = 13\), then \(1 / 2 = 7\) (and \(7 \cdot 2 = 1), 3 \cdot 5 = 2\), and so forth. using finite field arithmetic removes the need to worry about rounding errors and allows the system to work nicely with elliptic curves, which end up being necessary for the rest of the zk-snark machinery that makes the zk-snark protocol actually secure. special thanks to eran tromer for helping to explain many details about the inner workings of zk-snarks to me. secret sharing and erasure coding: a guide for the aspiring dropbox decentralizer | ethereum foundation blog ef blog search skip to contentcategories r&d research & development events events org organizational esp ecosystem support program eth.org ethereum.org sec security nxbn next billion protocol protocol announcements languages search secret sharing and erasure coding: a guide for the aspiring dropbox decentralizer posted by vitalik buterin on august 16, 2014 research & development one of the more exciting applications of decentralized computing that have aroused a considerable amount of interest in the past year is the concept of an incentivized decentralized online file storage system. currently, if you want your files or data securely backed up "in the cloud", you have three choices (1) upload them to your own servers, (2) use a centralized service like google drive or dropbox or (3) use an existing decentralized file system like freenet. these approaches all have their own faults; the first has a high setup and maintenance cost, the second relies on a single trusted party and often involves heavy price markups, and the third is slow and very limited in the amount of space that it allows each user because it relies on users to volunteer storage. incentivized file storage protocols have the potential to provide a fourth way, providing a much higher quantity of storage and quality of service by incentivizing actors to participate without introducing centralization. a number of platforms, including storj, maidsafe, to some extent permacoin, and filecoin, are attempting to tackle this problem, and the problem seems simple in the sense that all the tools are either already there or en route to being built, and all we need is the implementation. however, there is one part of the problem that is particularly important: how do we properly introduce redundancy? redundancy is crucial to security; especially in a decentralized network that will be highly populated by amateur and casual users, we absolutely cannot rely on any single node to stay online. we could simply replicate the data, having a few nodes each store a separate copy, but the question is: can we do better? as it turns out, we absolutely can. merkle trees and challenge-response protocols before we get into the nitty gritty of redundancy, we will first cover the easier part: how do we create at least a basic system that will incentivize at least one party to hold onto a file? without incentivization, the problem is easy; you simply upload the file, wait for other users to download it, and then when you need it again you can make a request querying for the file by hash. if we want to introduce incentivization, the problem becomes somewhat harder but, in the grand scheme of things, still not too hard. in the context of file storage, there are two kinds of activities that you can incentivize. the first is the actual act of sending the file over to you when you request it. this is easy to do; the best strategy is a simple tit-for-tat game where the sender sends over 32 kilobytes, you send over 0.0001 coins, the sender sends over another 32 kilobytes, etc. note that for very large files without redundancy this strategy is vulnerable to extortion attacks quite often, 99.99% of a file is useless to you without the last 0.01%, so the storer has the opportunity to extort you by asking for a very high payout for the last block. the cleverest fix to this problem is actually to make the file itself redundant, using a special kind of encoding to expand the file by, say, 11.11% so that any 90% of this extended file can be used to recover the original, and then hiding the exact redundancy percentage from the storer; however, as it turns out we will discuss an algorithm very similar to this for a different purpose later, so for now, simply accept that this problem has been solved. the second act that we can incentivize is the act of holding onto the file and storing it for the long term. this problem is somewhat harder how can you prove that you are storing a file without actually transferring the whole thing? fortunately, there is a solution that is not too difficult to implement, using what has now hopefully established a familiar reputation as the cryptoeconomist's best friend: merkle trees. well, patricia merkle might be better in some cases, to be precise. athough here the plain old original merkle will do. the basic approach is this. first, split the file up into very small chunks, perhaps somewhere between 32 and 1024 bytes each, and add chunks of zeroes until the number of chunks reaches n = 2^k for some k (the padding step is avoidable, but it makes the algorithm simpler to code and explain). then, we build the tree. rename the n chunks that we received chunk[n] to chunk[2n-1] , and then rebuild chunks 1 to n-1 with the following rule: chunk[i] = sha3([chunk[2*i], chunk[2*i+1]]) . this lets you calculate chunks n/2 to n-1 , then n/4 to n/2 1 , and so forth going up the tree until there is one "root", chunk[1] . now, note that if you store only the root, and forget about chunk[2] ... chunk[2n-1], the entity storing those other chunks can prove to you that they have any particular chunk with only a few hundred bytes of data. the algorithm is relatively simple. first, we define a function partner(n) which gives n-1 if n is odd, otherwise n+1 in short, given a chunk find the chunk that it is hashed together with in order to produce the parent chunk. then, if you want to prove ownership of chunk[k] with n <= k <= 2n-1 (ie. any part of the original file), submit chunk[partner(k)], chunk[partner(k/2)] (division here is assumed to round down, so eg. 11 / 2 = 5), chunk[partner(k/4)] and so on down to chunk[1], alongside the actual chunk[k]. essentially, we're providing the entire "branch" of the tree going up from that node all the way to the root. the verifier will then take chunk[k] and chunk[partner(k)] and use that to rebuild chunk[k/2], use that and chunk[partner(k/2)] to rebuild chunk[k/4] and so forth until the verifier gets to chunk[1], the root of the tree. if the root matches, then the proof is fine; otherwise it's not. the proof of chunk 10 includes (1) chunk 10, and (2) chunks 11 (11 = partner(10) ), 4 (4 = partner(10/2) ) and 3 (3 = partner(10/4) ). the verification process involves starting off with chunk 10, using each partner chunk in turn to recompute first chunk 5, then chunk 2, then chunk 1, and seeing if chunk 1 matches the value that the verifier had already stored as the root of the file. note that the proof implicitly includes the index sometimes you need to add the partner chunk on the right before hashing and sometimes on the left, and if the index used to verify the proof is different then the proof will not match. thus, if i ask for a proof of piece 422, and you instead provide even a valid proof of piece 587, i will notice that something is wrong. also, there is no way to provide a proof without ownership of the entire relevant section of the merkle tree; if you try to pass off fake data, at some point the hashes will mismatch and the final root will be different. now, let's go over the protocol. i construct a merkle tree out of the file as described above, and upload this to some party. then, every 12 hours, i pick a random number in [0, 2^k-1] and submit that number as a challenge. if the storer replies back with a merkle tree proof, then i verify the proof and if it is correct send 0.001 btc (or eth, or storjcoin, or whatever other token is used). if i receive no proof or an invalid proof, then i do not send btc. if the storer stores the entire file, they will succeed 100% of the time, if they store 50% of the file they will succeed 50% of the time, etc. if we want to make it all-or-nothing, then we can simply require the storer to solve ten consecutive proofs in order to get a reward. the storer can still get away with storing 99%, but then we take advantage of the same redundant coding strategy that i mentioned above and will describe below to make 90% of the file sufficient in any case. one concern that you may have at this point is privacy if you use a cryptographic protocol to let any node get paid for storing your file, would that not mean that your files are spread around the internet so that anyone can potentially access them? fortunately the answer to this is simple: encrypt the file before sending it out. from this point on, we'll assume that all data is encrypted, and ignore privacy because the presence of encryption resolves that issue almost completely (the "almost" being that the size of the file, and the times at which you access the file, are still public). looking to decentralize so now we have a protocol for paying people to store your data; the algorithm can even be made trust-free by putting it into an ethereum contract, using block.prevhash as a source of random data to generate the challenges. now let's go to the next step: figuring out how to decentralize the storage and add redundancy. the simplest way to decentralize is simple replication: instead of one node storing one copy of the file, we can have five nodes storing one copy each. however, if we simply follow the naive protocol above, we have a problem: one node can pretend to be five nodes and collect a 5x return. a quick fix to this is to encrypt the file five times, using five different keys; this makes the five identical copies indistinguishable from five different files, so a storer will not be able to notice that the five files are the same and store them once but claim a 5x reward. but even here we have two problems. first, there is no way to verify that the five copies of the file are stored by five separate users. if you want to have your file backed up by a decentralized cloud, you are paying for the service of decentralization; it makes the protocol have much less utility if all five users are actually storing everything through google and amazon. this is actually a hard problem; although encrypting the file five times and pretending that you are storing five different files will prevent a single actor from collecting a 5x reward with 1x storage, it cannot prevent an actor from collecting a 5x reward with 5x storage, and economies of scale mean even that situation will be desirable from the point of view of some storers. second, there is the issue that you are taking a large overhead, and especially taking the false-redundancy issue into account you are really not getting that much redundancy from it for example, if a single node has a 50% chance of being offline (quite reasonable if we're talking about a network of files being stored in the spare space on people's hard drives), then you have a 3.125% chance at any point that the file will be inaccessible outright. there is one solution to the first problem, although it is imperfect and it's not clear if the benefits are worth it. the idea is to use a combination of proof of stake and a protocol called "proof of custody" proof of simultaneous possession of a file and a private key. if you want to store your file, the idea is to randomly select some number of stakeholders in some currency, weighting the probability of selection by the number of coins that they have. implementing this in an ethereum contract might involve having participants deposit ether in the contract (remember, deposits are trust-free here if the contract provides a way to withdraw) and then giving each account a probability proportional to its deposit. these stakeholders will then receive the opportunity to store the file. then, instead of the simple merkle tree check described in the previous section, the proof of custody protocol is used. the proof of custody protocol has the benefit that it is non-outsourceable there is no way to put the file onto a server without giving the server access to your private key at the same time. this means that, at least in theory, users will be much less inclined to store large quantities of files on centralized "cloud" computing systems. of course, the protocol accomplishes this at the cost of much higher verification overhead, so that leaves open the question: do we want the verification overhead of proof of custody, or the storage overhead of having extra redundant copies just in case? m of n regardless of whether proof of custody is a good idea, the next step is to see if we can do a little better with redundancy than the naive replication paradigm. first, let's analyze how good the naive replication paradigm is. suppose that each node is available 50% of the time, and you are willing to take 4x overhead. in those cases, the chance of failure is 0.5 ^ 4 = 0.0625 a rather high value compared to the "four nines" (ie. 99.99% uptime) offered by centralized services (some centralized services offer five or six nines, but purely because of talebian black swan considerations any promises over three nines can generally be considered bunk; because decentralized networks do not depend on the existence or actions of any specific company or hopefully any specific software package, however, decentralized systems arguably actually can promise something like four nines legitimately). if we assume that the majority of the network will be quasi-professional miners, then we can reduce the unavailability percentage to something like 10%, in which case we actually do get four nines, but it's better to assume the more pessimistic case. what we thus need is some kind of m-of-n protocol, much like multisig for bitcoin. so let's describe our dream protocol first, and worry about whether it's feasible later. suppose that we have a file of 1 gb, and we want to "multisig" it into a 20-of-60 setup. we split the file up into 60 chunks, each 50 mb each (ie. 3 gb total), such that any 20 of those chunks suffice to reconstruct the original. this is information-theoretically optimal; you can't reconstruct a gigabyte out of less than a gigabyte, but reconstructing a gigabyte out of a gigabyte is entirely possible. if we have this kind of protocol, we can use it to split each file up into 60 pieces, encrypt the 60 chunks separately to make them look like independent files, and use an incentivized file storage protocol on each one separately. now, here comes the fun part: such a protocol actually exists. in this next part of the article, we are going to describe a piece of math that is alternately called either "secret sharing" or "erasure coding" depending on its application; the algorithm used for both those names is basically the same with the exception of one implementation detail. to start off, we will recall a simple insight: two points make a line. particularly, note that there is exactly one line that passes through those two points, and yet there is an infinite number of lines that pass through one point (and an infinite number of lines that pass through zero points). out of this simple insight, we can make a restricted 2-of-n version of our encoding: treat the first half of the file as the y coordinate of a line at x = 1 and the second half as the y coordinate of the line at x = 2 , draw the line, and take points at x = 3 , x = 4 , etc. any two pieces can then be used to reconstruct the line, and from there derive the y coordinates at x = 1 and x = 2 to get the file back. mathematically, there are two ways of doing this. the first is a relatively simple approach involving a system of linear equations. suppose that we file we want to split up is the number "1321". the left half is 13, the right half is 21, so the line joins (1, 13) and (2, 21). if we want to determine the slope and y-intercept of the line, we can just solve the system of linear equations: subtract the first equation from the second, and you get: and then plug that into the first equation, and get: so we have our equation, y = 8 * x + 5. we can now generate new points: (3, 29), (4, 37), etc. and from any two of those points we can recover the original equation. now, let's go one step further, and generalize this into m-of-n. as it turns out, it's more complicated but not too difficult. we know that two points make a line. we also know that three points make a parabola: thus, for 3-of-n, we just split the file into three, take a parabola with those three pieces as the y coordinates at x = 1, 2, 3 , and take further points on the parabola as additional pieces. if we want 4-of-n, we use a cubic polynomial instead. let's go through that latter case; we still keep our original file, "1321", but we'll split it up using 4-of-7 instead. our four points are (1, 1) , (2, 3) , (3, 2) , (4, 1) . so we have: eek! well, let's, uh, start subtracting. we'll subtract equation 1 from equation 2, 2 from 3, and 3 from 4, to reduce four equations to three, and then repeat that process again and again. so a = 1/2. now, we unravel the onion, and get: so b = -9/2, and then: so c = 12, and then: so a = 0.5, b = -4.5, c = 12, d = -7. here's the lovely polynomial visualized: i created a python utility to help you do this (this utility also does other more advanced stuff, but we'll get into that later); you can download it here. if you wanted to solve the equations quickly, you would just type in: > import share > share.sys_solve([[1.0, 1.0, 1.0, 1.0, -1.0], [8.0, 4.0, 2.0, 1.0, -3.0], [27.0, 9.0, 3.0, 1.0, -2.0], [64.0, 16.0, 4.0, 1.0, -1.0]]) [0.5, -4.5, 12.0, -7.0] note that putting the values in as floating point is necessary; if you use integers python's integer division will screw things up. now, we'll cover the easier way to do it, lagrange interpolation. the idea here is very clever: we come up with a cubic polynomial whose value is 1 at x = 1 and 0 at x = 2, 3, 4, and do the same for every other x coordinate. then, we multiply and add the polynomials together; for example, to match (1, 3, 2, 1) we simply take 1x the polynomial that passes through (1, 0, 0, 0), 3x the polynomial through (0, 1, 0, 0), 2x the polynomial through (0, 0, 1, 0) and 1x the polynomial through (0, 0, 0, 1) and then add those polynomials together to get the polynomal through (1, 3, 2, 1) (note that i said the polynomial passing through (1, 3, 2, 1); the trick works because four points define a cubic polynomial uniquely). this might not seem easier, because the only way we have of fitting polynomials to points to far is the cumbersome procedure above, but fortunately, we actually have an explicit construction for it: at x = 1, notice that the top and bottom are identical, so the value is 1. at x = 2, 3, 4, however, one of the terms on the top is zero, so the value is zero. multiplying up the polynomials takes quadratic time (ie. ~16 steps for 4 equations), whereas our earlier procedure took cubic time (ie. ~64 steps for 4 equations), so it's a substantial improvement especially once we start talking about larger splits like 20-of-60. the python utility supports this algorithm too: > import share > share.lagrange_interp([1.0, 3.0, 2.0, 1.0], [1.0, 2.0, 3.0, 4.0]) [-7.0, 12.000000000000002, -4.5, 0.4999999999999999] the first argument is the y coordinates, the second is the x coordinates. note the opposite order here; the code in the python module puts the lower-order coefficients of the polynomial first. and finally, let's get our additional shares: > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 5) 3.0 > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 6) 11.0 > share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], 7) 28.0 so here immediately we can see two problems. first, it looks like computerized floating point numbers aren't infinitely precise after all; the 12 turned into 12.000000000000002. second, the chunks start getting large as we move further out; at x = 10, it goes up to 163. this is somewhat breaking the promise that the amount of data you need to recover the file is the same size as the original file; if we lose x = 1, 2, 3, 4 then you need 8 digits to get the original values back and not 4. these are both serious issues, and ones that we will resolve with some more mathematical cleverness later, but we'll leave them aside for now. even with those issues remaining, we have basically achieved victory, so let's calculate our spoils. if we use a 20-of-60 split, and each node is online 50% of the time, then we can use combinatorics specifically, the binomial distribution formula to compute the probability that our data is okay. first, to set things up: > def fac(n): return 1 if n==0 else n * fac(n-1) > def choose(n,k): return fac(n) / fac(k) / fac(n-k) > def prob(n,k,p): return choose(n,k) * p ** k * (1-p) ** (n-k) the last formula computes the probability that exactly k servers out of n will be online if each individual server has a probability p of being online. now, we'll do: > sum([prob(60, k, 0.5) for k in range(0, 20)]) 0.0031088013296633353 99.7% uptime with only 3x redundancy a good step up from the 87.5% uptime that 3x redundancy would have given us had simple replication been the only tool in our toolkit. if we crank the redundancy up to 4x, then we get six nines, and we can stop there because the probability either ethereum or the entire internet will crash outright is greater than 0.0001% anyway (in fact, you're more likely to die tomorrow). oh, and if we assume each machine has 90% uptime (ie. hobbyist "farmers"), then with a 1.5x-redundant 20-of-30 protocol we get an absolutely overkill twelve nines. reputation systems can be used to keep track of how often each node is online. dealing with errors we'll spend the rest of this article discussing three extensions to this scheme. the first is a concern that you may have skipped over reading the above description, but one which is nonetheless important: what happens if some node tries to actively cheat? the algorithm above can recover the original data of a 20-of-60 split from any 20 pieces, but what if one of the data providers is evil and tries to provide fake data to screw with the algorithm. the attack vector is a rather compelling one: > share.lagrange_interp([1.0, 3.0, 2.0, 5.0], [1.0, 2.0, 3.0, 4.0]) [-11.0, 19.333333333333336, -8.5, 1.1666666666666665] taking the four points of the above polynomial, but changing the last value to 5, gives a completely different result. there are two ways of dealing with this problem. one is the obvious way, and the other is the mathematically clever way. the obvious way is obvious: when splitting a file, keep the hash of each chunk, and compare the chunk against the hash when receiving it. chunks that do not match their hashes are to be discarded. the clever way is somewhat more clever; it involves some spooky not-quite-moon-math called the berlekamp-welch algorithm. the idea is that instead of fitting just one polynomial, p, we imagine into existence two polynomials, q and e, such that q(x) = p(x) * e(x), and try to solve for both q and e at the same time. then, we compute p = q / e. the idea is that if the equation holds true, then for all x either p(x) = q(x) / e(x) or e(x) = 0; hence, aside from computing the original polynomial we magically isolate what the errors are. i won't go into an example here; the wikipedia article has a perfectly decent one, and you can try it yourself with: > map(lambda x: share.eval_poly_at([-7.0, 12.0, -4.5, 0.5], x), [1, 2, 3, 4, 5, 6]) [1.0, 3.0, 2.0, 1.0, 3.0, 11.0] > share.berlekamp_welch_attempt([1.0, 3.0, 18018.0, 1.0, 3.0, 11.0], [1, 2, 3, 4, 5, 6], 3) [-7.0, 12.0, -4.5, 0.5] > share.berlekamp_welch_attempt([1.0, 3.0, 2.0, 1.0, 3.0, 0.0], [1, 2, 3, 4, 5, 6], 3) [-7.0, 12.0, -4.5, 0.5] now, as i mentioned, this mathematical trickery is not really all that needed for file storage; the simpler approach of storing hashes and discarding any piece that does not match the recorded hash works just fine. but it is incidentally quite useful for another application: self-healing bitcoin addresses. bitcoin has a base58check encoding algorithm, which can be used to detect when a bitcoin address has been mistyped and returns an error so you do not accidentally send thousands of dollars into the abyss. however, using what we know, we can actually do better and make an algorithm which not only detects mistypes but also actually corrects the errors on the fly. we don't use any kind of clever address encoding for ethereum because we prefer to encourage use of name registry-based alternatives, but if an address encoding scheme was demanded something like this could be used. finite fields now, we get back to the second problem: once our x coordinates get a little higher, the y coordinates start shooting off very quickly toward infinity. to solve this, what we are going to do is nothing short of completely redefining the rules of arithmetic as we know them. specifically, let's redefine our arithmetic operations as: a + b := (a + b) % 11 a b := (a b) % 11 a * b := (a * b) % 11 a / b := (a * b ** 9) % 11 that "percent" sign there is "modulo", ie. "take the remainder of dividing that vaue by 11", so we have 7 + 5 = 1 , 6 * 6 = 3 (and its corollary 3 / 6 = 6 ), etc. we are now only allowed to deal with the numbers 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. the surprising thing is that, even as we do this, all of the rules about traditional arithmetic still hold with our new arithmetic; (a * b) * c = a * (b * c) , (a + b) * c = (a * c) + (b * c) , a / b * b = a if b != 0 , (a^2 b^2) = (a b)*(a + b) , etc. thus, we can simply take the algebra behind our polynomial encoding that we used above, and transplant it over into the new system. even though the intuition of a polynomial curve is completely borked we're now dealing with abstract mathematical objects and not anything resembling actual points on a plane because our new algebra is self-consistent, the formulas still work, and that's what counts. > e = share.mkmoduloclass(11) > p = share.lagrange_interp(map(e, [1, 3, 2, 1]), map(e, [1, 2, 3, 4])) > p [4, 1, 1, 6] > map(lambda x: share.eval_poly_at(map(e, p), e(x)), range(1, 9)) [1, 3, 2, 1, 3, 0, 6, 2] > share.berlekamp_welch_attempt(map(e, [1, 9, 9, 1, 3, 0, 6, 2]), map(e, [1, 2, 3, 4, 5, 6, 7, 8]), 3) [4, 1, 1, 6] the "map(e, [v1, v2, v3]) " is used to convert ordinary integers into elements in this new field; the software library includes an implementation of our crazy modulo 11 numbers that interfaces with arithmetic operators seamlessly so we can simply swap them in (eg. print e(6) * e(6) returns 3 ). you can see that everything still works except that now, because our new definitions of addition, subtraction, multiplication and division always return integers in [0 ... 10] we never need to worry about either floating point imprecision or the numbers expanding as the x coordinate gets too high. now, in reality these relatively simple modulo finite fields are not what are usually used in error-correcting codes; the generally preferred construction is something called a galois field (technically, any field with a finite number of elements is a galois field, but sometimes the term is used specifically to refer to polynomial-based fields as we will describe here). the idea is that the elements in the field are now polynomials, where the coefficients are themselves values in the field of integers modulo 2 (ie. a + b := (a + b) % 2, etc). adding and subtracting work as normally, but multiplying is itself modulo a polynomial, specifically x^8 + x^4 + x^3 + x + 1. this rather complicated multilayered construction lets us have a field with exactly 256 elements, so we can conveniently store every element in one byte and every byte as one element. if we want to work on chunks of many bytes at a time, we simply apply the scheme in parallel (ie. if each chunk is 1024 bytes, determine 10 polynomials, one for each byte, extend them separately, and combine the values at each x coordinate to get the chunk there). but it is not important to know the exact workings of this; the salient point is that we can redefine +, -, * and / in such a way that they are still fully self-consistent but always take and output bytes. going multidimensional: the self-healing cube now, we're using finite fields, and we can deal with errors, but one issue still remains: what happens when nodes do go down? at any point in time, you can count on 50% of the nodes storing your file staying online, but what you cannot count on is the same nodes staying online forever eventually, a few nodes are going to drop out, then a few more, then a few more, until eventually there are not enough of the original nodes left online. how do we fight this gradual attrition? one strategy is that you could simply watch the contracts that are rewarding each individual file storage instance, seeing when some stop paying out rewards, and then re-upload the file. however, there is a problem: in order to re-upload the file, you need to reconstruct the file in its entirety, a potentially difficult task for the multi-gigabyte movies that are now needed to satisfy people's seemingly insatiable desires for multi-thousand pixel resolution. additionally, ideally we would like the network to be able to heal itself without requiring active involvement from a centralized source, even the owner of the files. fortunately, such an algorithm exists, and all we need to accomplish it is a clever extension of the error correcting codes that we described above. the fundamental idea that we can rely on is the fact that polynomial error correcting codes are "linear", a mathematical term which basically means that it interoperates nicely with multiplication and addition. for example, consider: > share.lagrange_interp([1.0, 3.0, 2.0, 1.0], [1.0, 2.0, 3.0, 4.0]) [-7.0, 12.000000000000002, -4.5, 0.4999999999999999] > share.lagrange_interp([10.0, 5.0, 5.0, 10.0], [1.0, 2.0, 3.0, 4.0]) [20.0, -12.5, 2.5, 0.0] > share.lagrange_interp([11.0, 8.0, 7.0, 11.0], [1.0, 2.0, 3.0, 4.0]) [13.0, -0.5, -2.0, 0.5000000000000002] > share.lagrange_interp([22.0, 16.0, 14.0, 22.0], [1.0, 2.0, 3.0, 4.0]) [26.0, -1.0, -4.0, 1.0000000000000004] see how the input to the third interpolation is the sum of the inputs to the first two, and the output ends up being the sum of the first two outputs, and then when we double the input it also doubles the output. so what's the benefit of this? well, here's the clever trick. erasure cording is itself a linear formula; it relies only on multiplication and addition. hence, we are going to apply erasure coding to itself. so how are we going to do this? here is one possible strategy. first, we take our 4-digit "file" and put it into a 2x2 grid. 1 3 2 1 then, we use the same polynomial interpolation and extension process as above to extend the file along both the x and y axes: 1 3 5 7 2 1 0 10 3 10 4 8 and then we apply the process again to get the remaining 4 squares: 1 3 5 7 2 1 0 10 3 10 6 2 4 8 1 5 note that it doesn't matter if we get the last four squares by expanding horizontally and vertically; because secret sharing is linear it is commutative with itself, so you get the exact same answer either way. now, suppose we lose a number in the middle, say, 6. well, we can do a repair vertically: > share.repair([5, 0, none, 1], e) [5, 0, 6, 1] or horizontally: > share.repair([3, 10, none, 2], e) [3, 10, 6, 2] and tada, we get 6 in both cases. this is the surprising thing: the polynomials work equally well on both the x or the y axis. hence, if we take these 16 pieces from the grid, and split them up among 16 nodes, and one of the nodes disappears, then nodes along either axis can come together and reconstruct the data that was held by that particular node and start claiming the reward for storing that data. ideally, we can even extend this process beyond 2 dimensions, producing a 3-dimensional cube, a 4-dimensional hypercube or more the gain of using more dimensions is ease of reconstruction, and the cost is a lower degree of redundancy. thus, what we have is an information-theoretic equivalent of something that sounds like it came straight out of science-fiction: a highly redundant, interlinking, modular self-healing cube, that can quickly locally detect and fix its own errors even if large sections of the cube were to be damaged, co-opted or destroyed. "the cube can still function even if up to 78% of it were to be destroyed..." so, let's put it all together. you have a 10 gb file, and you want to split it up across the network. first, you encrypt the file, and then you split the file into, let's say, 125 chunks. you arrange these chunks into a 3-dimensional 5x5x5 cube, figure out the polynomial along each axis, and "extend" each one so that at the end you have a 7x7x7 cube. you then look for 343 nodes willing to store each piece of data, and tell each node only the identity of the other nodes that are along the same axis (we want to make an effort to avoid a single node gathering together an entire line, square or cube and storing it and calculating any redundant chunks as needed in real-time, getting the reward for storing all the chunks of the file without actually providing any redundancy. in order to actually retrieve the file, you would send out a request for all of the chunks, then see which of the pieces coming in have the highest bandwidth. you may use the pay-per-chunk protocol to pay for the sending of the data; extortion is not an issue because you have such high redundancy so no one has the monopoly power to deny you the file. as soon as the minimal number of pieces arrive, you would do the math to decrypt the pieces and reconstitute the file locally. perhaps, if the encoding is per-byte, you may even be able to apply this to a youtube-like streaming implementation, reconstituting one byte at a time. in some sense, there is an unavoidable tradeoff between self-healing and vulnerability to this kind of fake redundancy: if parts of the network can come together and recover a missing piece to provide redundancy, then a malicious large actor in the network can recover a missing piece on the fly to provide and charge for fake redundancy. perhaps some scheme involving adding another layer of encryption on each piece, hiding the encryption keys and the addresses of the storers of the individual pieces behind yet another erasure code, and incentivizing the revelation process only at some particular times might form an optimal balance. secret sharing at the beginning of the article, i mentioned another name for the concept of erasure coding, "secret sharing". from the name, it's easy to see how the two are related: if you have an algorithm for splitting data up among 9 nodes such that 5 of 9 nodes are needed to recover it but 4 of 9 can't, then another obvious use case is to use the same algorithm for storing private keys split up your bitcoin wallet backup into nine parts, give one to your mother, one to your boss, one to your lawyer, put three into a few safety deposit boxes, etc, and if you forget your password then you'll be able to ask each of them individually and chances are at least five will give you your pieces back, but the individuals themselves are sufficiently far apart from each other that they're unlikely to collude with each other. this is a very legitimate thing to do, but there is one implementation detail involved in doing it right. the issue is this: even though 4 of 9 can't recover the original key, 4 of 9 can still come together and have quite a lot of information about it specifically, four linear equations over five unknowns. this reduces the dimensionality of the choice space by a factor of 5, so instead of 2256 private keys to search through they now have only 251. if your key is 180 bits, that goes down to 236 trivial work for a reasonably powerful computer. the way we fix this is by erasure-coding not just the private key, but rather the private key plus 4x as many bytes of random gook. more precisely, let the private key be the zero-degree coefficient of the polynomial, pick four random values for the next four coefficients, and take values from that. this makes each piece five times longer, but with the benefit that even 4 of 9 now have the entire choice space of 2180 or 2256 to search through. conclusion so there we go, that's an introduction to the power of erasure coding arguably the single most underhyped set of algorithms (except perhaps scip) in computer science or cryptography. the ideas here essentially are to file storage what multisig is to smart contracts, allowing you to get the absolutely maximum possible amount of security and redundancy out of whatever ratio of storage overhead you are willing to accept. it's an approach to file storage availability that strictly supersedes the possibilities offered by simple splitting and replication (indeed, replication is actually exactly what you get if you try to apply the algorithm with a 1-of-n strategy), and can be used to encapsulate and separately handle the problem of redundancy in the same way that encryption encapsulates and separately handles the problem of privacy. decentralized file storage is still far from a solved problem; although much of the core technology, including erasure coding in tahoe-lafs, has already been implemented, there are certainly many minor and not-so-minor implementation details that still need to be solved for such a setup to actually work. an effective reputation system will be required for measuring quality-of-service (eg. a node up 99% of the time is worth at least 3x more than a node up 50% of the time). in some ways, incentivized file storage even depends on effective blockchain scalability; having to implicitly pay for the fees of 343 transactions going to verification contracts every hour is not going to work until transaction fees become far lower than they are today, and until then some more coarse-grained compromises are going to be required. but then again, pretty much every problem in the cryptocurrency space still has a very long way to go. previous post next post subscribe to protocol announcements sign up to receive email notifications for protocol-related announcements, such as network upgrades, faqs or security issues. you can opt-out of these at any time. sign up ethereum foundation • ethereum.org • esp • bug bounty program • do-not-track • archive categories research & development • events • organizational • ecosystem support program • ethereum.org • security • next billion • protocol announcements dark mode toggle analyzing token sale models 2017 jun 09 see all posts note: i mention the names of various projects below only to compare and contrast their token sale mechanisms; this should not be taken as an endorsement or criticism of any specific project as a whole. it's entirely possible for any given project to be total trash as a whole and yet still have an awesome token sale model. the last few months have seen an increasing amount of innovation in token sale models. two years ago, the space was simple: there were capped sales, which sold a fixed number of tokens at a fixed price and hence fixed valuation and would often quickly sell out, and there were uncapped sales, which sold as many tokens as people were willing to buy. now, we have been seeing a surge of interest, both in terms of theoretical investigation and in many cases real-world implementation, of hybrid capped sales, reverse dutch auctions, vickrey auctions, proportional refunds, and many other mechanisms. many of these mechanisms have arisen as responses to perceived failures in previous designs. nearly every significant sale, including brave's basic attention tokens, gnosis, upcoming sales such as bancor, and older ones such as maidsafe and even the ethereum sale itself, has been met with a substantial amount of criticism, all of which points to a simple fact: so far, we have still not yet discovered a mechanism that has all, or even most, of the properties that we would like. let us review a few examples. maidsafe the decentralized internet platform raised $7m in five hours. however, they made the mistake of accepting payment in two currencies (btc and msc), and giving a favorable rate to msc buyers. this led to a temporary ~2x appreciation in the msc price, as users rushed in to buy msc to participate in the sale at the more favorable rate, but then the price saw a similarly steep drop after the sale ended. many users converted their btc to msc to participate in the sale, but then the sale closed too quickly for them, leading to them being stuck with a ~30% loss. this sale, and several others after it (cough cough wetrust, tokencard), showed a lesson that should hopefully by now be uncontroversial: running a sale that accepts multiple currencies at a fixed exchange rate is dangerous and bad. don't do it. ethereum the ethereum sale was uncapped, and ran for 42 days. the sale price was 2000 eth for 1 btc for the first 14 days, and then started increasing linearly, finishing at 1337 eth for 1 btc. nearly every uncapped sale is criticized for being "greedy" (a criticism i have significant reservations about, but we'll get back to this later), though there is also another more interesting criticism of these sales: they give participants high uncertainty about the valuation that they are buying at. to use a not-yet-started sale as a example, there are likely many people who would be willing to pay $10,000 for a pile of bancor tokens if they knew for a fact that this pile represented 1% of all bancor tokens in existence, but many of them would become quite apprehensive if they were buying a pile of, say, 5000 bancor tokens, and they had no idea whether the total supply would be 50000, 500000 or 500 million. in the ethereum sale, buyers who really cared about predictability of valuation generally bought on the 14th day, reasoning that this was the last day of the full discount period and so on this day they had maximum predictability together with the full discount, but the pattern above is hardly economically optimal behavior; the equilibrium would be something like everyone buying in on the last hour of the 14th day, making a private tradeoff between certainty of valuation and taking the 1.5% hit (or, if certainty was really important, purchases could spill over into the 15th, 16th and later days). hence, the model certainly has some rather weird economic properties that we would really like to avoid if there is a convenient way to do so. bat throughout 2016 and early 2017, the capped sale design was most popular. capped sales have the property that it is very likely that interest is oversubscribed, and so there is a large incentive to getting in first. initially, sales took a few hours to finish. however, soon the speed began to accelerate. first blood made a lot of news by finishing their $5.5m sale in two minutes while active denial-of-service attacks on the ethereum blockchain were taking place. however, the apotheosis of this race-to-the-nash-equilibrium did not come until the bat sale last month, when a $35m sale was completed within 30 seconds due to the large amount of interest in the project. not only did the sale finish within two blocks, but also: the total transaction fees paid were 70.15 eth (>$15,000), with the highest single fee being ~$6,600 185 purchases were successful, and over 10,000 failed the ethereum blockchain's capacity was full for 3 hours after the sale started thus, we are starting to see capped sales approach their natural equilibrium: people trying to outbid each other's transaction fees, to the point where potentially millions of dollars of surplus would be burned into the hands of miners. and that's before the next stage starts: large mining pools butting into the start of the line and just buying up all of the tokens themselves before anyone else can. gnosis the gnosis sale attempted to alleviate these issues with a novel mechanism: the reverse dutch auction. the terms, in simplified form, are as follows. there was a capped sale, with a cap of $12.5 million usd. however, the portion of tokens that would actually be given to purchasers depended on how long the sale took to finish. if it finished on the first day, then only ~5% of tokens would be distributed among purchasers, and the rest held by the gnosis team; if it finished on the second day, it would be ~10%, and so forth. the purpose of this is to create a scheme where, if you buy at time \(t\), then you are guaranteed to buy in at a valuation which is at most \(\frac{1}{t}\). the goal is to create a mechanism where the optimal strategy is simple. first, you personally decide what is the highest valuation you would be willing to buy at (call it v). then, when the sale starts, you don't buy in immediately; rather, you wait until the valuation drops to below that level, and then send your transaction. there are two possible outcomes: the sale closes before the valuation drops to below v. then, you are happy because you stayed out of what you thought is a bad deal. the sale closes after the valuation drops to below v. then, you sent your transaction, and you are happy because you got into what you thought is a good deal. however, many people predicted that because of "fear of missing out" (fomo), many people would just "irrationally" buy in at the first day, without even looking at the valuation. and this is exactly what happened: the sale finished in a few hours, with the result that the sale reached its cap of $12.5 million when it was only selling about 5% of all tokens that would be in existence an implied valuation of over $300 million. all of this would of course be an excellent piece of confirming evidence for the narrative that markets are totally irrational, people don't think clearly before throwing in large quantities of money (and often, as a subtext, that the entire space needs to be somehow suppressed to prevent further exuberance) if it weren't for one inconvenient fact: the traders who bought into the sale were right. even in eth terms, despite the massive eth price rise, the price of 1 gno has increased from ~0.6 eth to ~0.8 eth. what happened? a couple of weeks before the sale started, facing public criticism that if they end up holding the majority of the coins they would act like a central bank with the ability to heavily manipulate gno prices, the gnosis team agreed to hold 90% of the coins that were not sold for a year. from a trader's point of view, coins that are locked up for a long time are coins that cannot affect the market, and so in a short term analysis, might as well not exist. this is what initially propped up steem to such a high valuation last year in july, as well as zcash in the very early moments when the price of each coin was over $1,000. now, one year is not that long a time, and locking up coins for a year is nowhere close to the same thing as locking them up forever. however, the reasoning goes further. even after the one year holding period expires, you can argue that it is in the gnosis team's interest to only release the locked coins if they believe that doing so will make the price go up, and so if you trust the gnosis team's judgement this means that they are going to do something which is at least as good for the gno price as simply locking up the coins forever. hence, in reality, the gno sale was really much more like a capped sale with a cap of $12.5 million and a valuation of $37.5 million. and the traders who participated in the sale reacted exactly as they should have, leaving scores of internet commentators wondering what just happened. there is certainly a weird bubbliness about crypto-assets, with various no-name assets attaining market caps of $1-100 million (including bitbean as of the time of this writing at $12m, potcoin at $22m, pepecash at $13m and smileycoin at $14.7m) just because. however, there's a strong case to be made that the participants at the sale stage are in many cases doing nothing wrong, at least for themselves; rather, traders who buy in sales are simply (correctly) predicting the existence of an ongoing bubble has been brewing since the start of 2015 (and arguably, since the start of 2010). more importantly though, bubble behavior aside, there is another legitimate criticism of the gnosis sale: despite their 1-year no-selling promise, eventually they will have access to the entirety of their coins, and they will to a limited extent be able to act like a central bank with the ability to heavily manipulate gno prices, and traders will have to deal with all of the monetary policy uncertainty that that entails. specifying the problem so what would a good token sale mechanism look like? one way that we can start off is by looking through the criticisms of existing sale models that we have seen and coming up with a list of desired properties. let's do that. some natural properties include: certainty of valuation if you participate in a sale, you should have certainty over at least a ceiling on the valuation (or, in other words, a floor on the percentage of all tokens you are getting). certainty of participation if you try to participate in a sale, you should be able to generally count on succeeding. capping the amount raised to avoid being perceived as greedy (or possibly to mitigate risk of regulatory attention), the sale should have a limit on the amount of money it is collecting. no central banking the token sale issuer should not be able to end up with an unexpectedly very large percentage of the tokens that would give them control over the market. efficiency the sale should not lead to substantial economic inefficiencies or deadweight losses. sounds reasonable? well, here's the not-so-fun part. (1) and (2) cannot be fully satisfied simultaneously. at least without resorting to very clever tricks, (3), (4) and (5) cannot be satisfied simultaneously. these can be cited as "the first token sale dilemma" and "the second token sale trilemma". the proof for the first dilemma is easy: suppose you have a sale where you provide users with certainty of a $100 million valuation. now, suppose that users try to throw $101 million into the sale. at least some will fail. the proof for the second trilemma is a simple supply-and-demand argument. if you satisfy (4), then you are selling all, or some fixed large percentage, of the tokens, and so the valuation you are selling at is proportional to the price you are selling at. if you satisfy (3), then you are putting a cap on the price. however, this implies the possibility that the equilibrium price at the quantity you are selling exceeds the price cap that you set, and so you get a shortage, which inevitably leads to either (i) the digital equivalent of standing in line for 4 hours at a very popular restaurant, or (ii) the digital equivalent of ticket scalping both large deadwight losses, contradicting (5). the first dilemma cannot be overcome; some valuation uncertainty or participation uncertainty is inescapable, though when the choice exists it seems better to try to choose participation uncertainty rather than valuation uncertainty. the closest that we can come is compromising on full participation to guarantee partial participation. this can be done with a proportional refund (eg. if $101 million buy in at a $100 million valuation, then everyone gets a 1% refund). we can also think of this mechanism as being an uncapped sale where part of the payment comes in the form of locking up capital rather than spending it; from this viewpoint, however, it becomes clear that the requirement to lock up capital is an efficiency loss, and so such a mechanism fails to satisfy (5). if ether holdings are not well-distributed then it arguably harms fairness by favoring wealthy stakeholders. the second dilemma is difficult to overcome, and many attempts to overcome it can easily fail or backfire. for example, the bancor sale is considering limiting the transaction gas price for purchases to 50 shannon (~12x the normal gasprice). however, this now means that the optimal strategy for a buyer is to set up a large number of accounts, and from each of those accounts send a transaction that triggers a contract, which then attempts to buy in (the indirection is there to make it impossible for the buyer to accidentally buy in more than they wanted, and to reduce capital requirements). the more accounts a buyer sets up, the more likely they are to get in. hence, in equilibrium, this could lead to even more clogging of the ethereum blockchain than a bat-style sale, where at least the $6600 fees were spent on a single transaction and not an entire denial-of-service attack on the network. furthermore, any kind of on-chain transaction spam contest severely harms fairness, because the cost of participating in the contest is constant, whereas the reward is proportional to how much money you have, and so the result disproportionately favors wealthy stakeholders. moving forward there are three more clever things that you can do. first, you can do a reverse dutch auction just like gnosis, but with one change: instead of holding the unsold tokens, put them toward some kind of public good. simple examples include: (i) airdrop (ie. redistributing to all eth holders), (ii) donating to the ethereum foundation, (iii) donating to parity, brainbot, smartpool or other companies and individuals independently building infrastructure for the ethereum space, or (iv) some combination of all three, possibly with the ratios somehow being voted on by the token buyers. second, you can keep the unsold tokens, but solve the "central banking" problem by committing to a fully automated plan for how they would be spent. the reasoning here is similar to that for why many economists are interested in rules-based monetary policy: even if a centralized entity has a large amount of control over a powerful resource, much of the political uncertainty that results can be mitigated if the entity credibly commits to following a set of programmatic rules for how they apply it. for example, the unsold tokens can be put into a market maker that is tasked with preserving the tokens' price stability. third, you can do a capped sale, where you limit the amount that can be bought by each person. doing this effectively requires a kyc process, but the nice thing is a kyc entity can do this once, whitelisting users' addresses after they verify that the address represents a unique individual, and this can then be reused for every token sale, alongside other applications that can benefit from per-person sybil resistance like akasha's quadratic voting. there is still deadweight loss (ie. inefficiency) here, because this will lead to individuals with no personal interest in tokens participating in sales because they know they will be able to quickly flip them on the market for a profit. however, this is arguably not that bad: it creates a kind of crypto universal basic income, and if behavioral economics assumptions like the endowment effect are even slightly true it will also succeed at the goal of ensuring widely distributed ownership. are single round sales even good? let us get back to the topic of "greed". i would claim that not many people are, in principle, opposed to the idea of development teams that are capable of spending $500 million to create a really great project getting $500 million. rather, what people are opposed to is (i) the idea of completely new and untested development teams getting $50 million all at once, and (ii) even more importantly, the time mismatch between developers' rewards and token buyers' interests. in a single-round sale, the developers have only one chance to get money to build the project, and that is near the start of the development process. there is no feedback mechanism where teams are first given a small amount of money to prove themselves, and then given access to more and more capital over time as they prove themselves to be reliable and successful. during the sale, there is comparatively little information to filter between good development teams and bad ones, and once the sale is completed, the incentive to developers to keep working is relatively low compared to traditional companies. the "greed" isn't about getting lots of money, it's about getting lots of money without working hard to show you're capable of spending it wisely. if we want to strike at the heart of this problem, how would we solve it? i would say the answer is simple: start moving to mechanisms other than single round sales. i can offer several examples as inspiration: angelshares this project ran a sale in 2014 where it sold off a fixed percentage of all ags every day for a period of several months. during each day, people could contribute an unlimited amount to the crowdsale, and the ags allocation for that day would be split among all contributors. basically, this is like having a hundred "micro-rounds" of uncapped sales over the course of most of a year; i would claim that the duration of the sales could be stretched even further. mysterium, which held a little-noticed micro-sale six months before the big one. bancor, which recently agreed to put all funds raised over a cap into a market maker which will maintain price stability along with maintaining a price floor of 0.01 eth. these funds cannot be removed from the market maker for two years. it seems hard to see the relationship between bancor's strategy and solving time mismatch incentives, but an element of a solution is there. to see why, consider two scenarios. as a first case, suppose the sale raises $30 million, the cap is $10 million, but then after one year everyone agrees that the project is a flop. in this case, the price would try to drop below 0.01 eth, and the market maker would lose all of its money trying to maintain the price floor, and so the team would only have $10 million to work with. as a second case, suppose the sale raises $30 million, the cap is $10 million, and after two years everyone is happy with the project. in this case, the market maker will not have been triggered, and the team would have access to the entire $30 million. a related proposal is vlad zamfir's "safe token sale mechanism". the concept is a very broad one that could be parametrized in many ways, but one way to parametrize it is to sell coins at a price ceiling and then have a price floor slightly below that ceiling, and then allow the two to diverge over time, freeing up capital for development over time if the price maintains itself. arguably, none of the above three are sufficient; we want sales that are spread out over an even longer period of time, giving us much more time to see which development teams are the most worthwhile before giving them the bulk of their capital. but nevertheless, this seems like the most productive direction to explore in. coming out of the dilemmas from the above, it should hopefully be clear that while there is no way to counteract the dilemma and trilemma head on, there are ways to chip away at the edges by thinking outside the box and compromising on variables that are not apparent from a simplistic view of the problem. we can compromise on guarantee of participation slightly, mitigating the impact by using time as a third dimension: if you don't get in during round \(n\), you can just wait until round \(n+1\) which will be in a week and where the price probably will not be that different. we can have a sale which is uncapped as a whole, but which consists of a variable number of periods, where the sale within each period is capped; this way teams would not be asking for very large amounts of money without proving their ability to handle smaller rounds first. we can sell small portions of the token supply at a time, removing the political uncertainty that this entails by putting the remaining supply into a contract that continues to sell it automatically according to a prespecified formula. here are a few possible mechanisms that follow some of the spirit of the above ideas: host a gnosis-style reverse dutch auction with a low cap (say, $1 million). if the auction sells less than 100% of the token supply, automatically put the remaining funds into another auction two months later with a 30% higher cap. repeat until the entire token supply is sold. sell an unlimited number of tokens at a price of \(\$x\) and put 90% of the proceeds into a smart contract that guarantees a price floor of \(\$0.9 \cdot x\). have the price ceiling go up hyperbolically toward infinity, and the price floor go down linearly toward zero, over a five-year period. do the exact same thing angelshares did, though stretch it out over 5 years instead of a few months. host a gnosis-style reverse dutch auction. if the auction sells less than 100% of the token supply, put the remaining funds into an automated market maker that attempts to ensure the token's price stability (note that if the price continues going up anyway, then the market maker would be selling tokens, and some of these earnings could be given to the development team). immediately put all tokens into a market maker with parameters+variables \(x\) (minimum price), \(s\) (fraction of all tokens already sold), \(t\) (time since sale started), \(t\) (intended duration of sale, say 5 years), that sells tokens at a price of \(\dfrac{k}{(\frac{t}{t s})}\) (this one is weird and may need to be economically studied more). note that there are other mechanisms that should be tried to solve other problems with token sales; for example, revenues going into a multisig of curators, which only hand out funds if milestones are being met, is one very interesting idea that should be done more. however, the design space is highly multidimensional, and there are a lot more things that could be tried. dark mode toggle situazioni di collusione 2000 jan 01 see all posts un ringraziamento speciale a glen weyl, phil daian e jinglan wang per la revisione, e hacktar e matteopey per la traduzione. negli ultimi anni si è registrato un crescente interesse nell'utilizzo di incentivi economici e meccanismi deliberatamente progettati per allineare il comportamento dei partecipanti in vari contesti. per quanto riguarda la blockchain, la struttura del meccanismo fornisce prima di tutto la sicurezza per la blockchain stessa, incoraggia i miner o validatori proof of stake a partecipare onestamente, ma più di recente viene applicato in mercati di previsione, "registri selezionati tramite token" e molti altri contesti. il nascente movimento radicalxchange ha nel frattempo iniziato una sperimentazione con tasse harberg, voto quadratico, finanziamento quadratico e altro. recentemente si è anche registrato un crescente interesse a utilizzare incentivi basati su token per cercare di incoraggiare la diffusione di post di qualità sui social media. tuttavia, man mano che lo sviluppo di questi sistemi passa dalla teoria alla pratica, si presenta una serie di sfide, che secondo me non sono state ancora adeguatamente considerate. come esempio recente di questo passaggio dalla teoria alla pratica, bihu, una piattaforma cinese, ha da poco rilasciato un meccanismo basato su moneta per incoraggiare le persone a scrivere post. il meccanismo di base (potete consultare il white paper in cinese qui) è che se un utente della piattaforma detiene token key, ha la possibilità di impegnare quei token key in articoli; ogni utente ha a disposizione k "voti positivi" al giorno, e il "peso" di ogni voto è proporzionale alla quota dall'utente che vota. gli articoli con una maggiore quantità di quote di voti positivi appaiono in modo più prominente e l'autore di un articolo ottiene una ricompensa di token key più o meno proporzionale alla quantità di voti positivi key assegnati all'articolo. questa in realtà è una semplificazione eccessiva e il meccanismo attuale presenta alcune non linearità, che tuttavia non sono essenziali per il funzionamento di base del meccanismo. key ha valore perché può essere utilizzato in vari modi all'interno della piattaforma, ma in particolare una percentuale di tutte le entrate pubblicitarie viene utilizzata per acquistare e utilizzare key (meritano un grande applauso per aver ideato questo meccanismo e non aver realizzato l'ennesimo mezzo di scambio di token). questo tipo di progetto non è sicuramente unico al mondo; incentivare la creazione di contenuti online è qualcosa di cui si occupano in molti, e ci sono stati molti progetti simili, e altri piuttosto diversi. questa particolare piattaforma viene già utilizzata in modo significativo: alcuni mesi fa, il subreddit trading /r/ethtrader di ethereum ha introdotto una funzione sperimentale per certi versi simile, in cui un token chiamato "donut" viene erogato agli utenti che fanno commenti che vengono votati positivamente, con una quantità prestabilita di donut rilasciata settimanalmente agli utenti in proporzione al numero di voti positivi ricevuti dai loro commenti. i donut possono essere utilizzati per acquistare la possibilità di impostare il contenuto del banner nella parte superiore del subreddit, nonché per votare nei sondaggi della community. tuttavia, diversamente da ciò che accade nel sistema key, in questo caso la ricompensa che b riceve quando a assegna un voto positivo a b non è proporzionale alla fornitura di monete esistente di a; invece, ogni account reddit ha la stessa capacità di contribuire ad altri account reddit. questo tipo di esperimenti, che tentano di premiare la creazione di contenuti di qualità in un modo che va oltre i limiti noti di donazione/microtipping, sono molto importanti; un compenso insufficiente per la creazione di contenuti internet generati dagli utenti è un problema molto sentito nella società in generale (provate a leggere "liberal radicalism" e "data as labor"), ed è incoraggiante vedere che le crypto community tentano di usare il potere della progettazione per cercare una soluzione. ** ma sfortunatamente, questi sistemi sono anche vulnerabili agli attacchi. ** autovotazione, plutocrazia e corruzione ecco come si potrebbe attaccare economicamente il progetto proposto sopra. supponiamo che un utente facoltoso ottenga una quantità di token n e, di conseguenza, ciascuno dei k voti positivi dell'utente dia al destinatario una ricompensa di n * q (q qui probabilmente è un numero molto piccolo, ad esempio provate a pensare q = 0,000001). l'utente semplicemente assegna voti positivi ai propri account fake, assegnandosi autonomamente una ricompensa di n * k * q. quindi, il sistema semplicemente finisce per assegnare a ogni utente un "tasso di interesse" di k * q per periodo, e il meccanismo non ha altri scopi. il meccanismo di bihu, forse anticipando tutto ciò, ha una logica superlineare in cui gli articoli con più key con voti positivi ottengono una ricompensa sproporzionatamente maggiore, apparentemente per incoraggiare a votare positivamente per i post popolari piuttosto che votare per se stessi. è comune per i sistemi di governance con votazione a moneta aggiungere questo tipo di superlinearità per impedire che l'autovotazione metta a repentaglio l'intero sistema; la maggior parte degli schemi dpos ha un numero limitato di slot delegati con ricompensa zero per chiunque non riceva abbastanza voti per unirsi a uno degli slot, con effetti simili. questi sistemi, però, introducono inevitabilmente due nuovi punti deboli: incoraggiano la plutocrazia, in quanto individui e cartelli molto ricchi possono comunque ottenere fondi sufficienti per autovotarsi. possono essere aggirati da utenti che corrompono altri utenti per votarli in massa. gli attacchi di corruzione possono sembrare inverosimili (chi ha mai accettato una bustarella nella vita reale?), ma in un ecosistema maturo sono molto più realistici di quanto sembri. nella maggior parte dei contesti in cui la corruzione ha avuto luogo nell'ambito della blockchain, gli operatori usano un nuovo eufemismo per dare al concetto un volto amichevole: non è una bustarella, è un "pool di persone" che "condivide i dividendi". le tangenti possono anche essere mascherate: immaginate uno scambio di criptovalute che offre zero commissioni e si sforza di realizzare un'interfaccia utente eccezionalmente fruibile, e non cerca nemmeno di ottenere profitti. utilizza invece monete che gli utenti depositano per partecipare a diversi sistemi di voto con moneta. vi saranno inevitabilmente persone che considereranno normale la collusione all'interno dei gruppi: si veda ad esempio un recente scandalo che ha coinvolto eos dpos: infine, esiste la possibilità di una "bustarella negativa", vale a dire: ricatti o coercizioni, minacce di danneggiare i partecipanti se non agiscono all'interno del meccanismo in un determinato modo. nell'esperimento /r/ethtrader, la paura che le persone acquistassero donut per pilotare i sondaggi di governance ha portato la community a decidere di rendere solo i donut bloccati (cioè non scambiabili) idonei per le votazioni. c'è tuttavia un attacco ancora più a basso costo dell'acquisto di donut (un attacco che può essere considerato una specie di bustarella nascosta): il noleggio. se un hacker detiene già eth, può utilizzarli come garanzia su una piattaforma come compound per ottenere in prestito alcuni token, con il pieno diritto di utilizzare questi token per qualsiasi scopo, inclusa la partecipazione a voti, e al termine semplicemente rimandare i token al contratto di prestito per recuperare la garanzia. tutto avviene senza dover sopportare nemmeno un secondo di esposizione al prezzo del token che ha appena usato per un voto decisivo con moneta, anche se il meccanismo di votazione con moneta include un blocco temporale (come ad esempio in bihu). in ogni caso, le questioni relative a corruzione e al potere eccessivo che possono assumere partecipanti ricchi e con collegamenti importanti sono sorprendentemente difficili da evitare. identità alcuni sistemi tentano di mitigare gli aspetti plutocratici del voto con moneta utilizzando un sistema d'identità. nel caso del sistema /r/ethtrader con donut, ad esempio, anche se i sondaggi di governance sono effettuati tramite un voto con moneta, il meccanismo che determina quanti donut (cioè monete) si ottengono si basa sugli account reddit: 1 voto positivo da 1 account reddit = n donut guadagnati. l'obiettivo ideale di un sistema di identità è rendere relativamente facile per gli individui ottenere un'identità, ma relativamente difficile ottenerne molte. nel sistema di donut /r/ethtrader vengono utilizzati gli account reddit, nel corrispondente clr gitcoin si utilizzano allo stesso scopo gli account github. ma l'identità, almeno nel modo in cui è stata implementata finora, è qualcosa di fragile... siete troppo pigri per fare incetta di telefoni? allora forse vi serve questo: vale il solito avvertimento su come questi siti possano o meno essere truffaldini: fate le vostre ricerche e state attenti. è possibile che attaccare questi meccanismi semplicemente controllando migliaia di identità false, come dei burattinai, sia persino più facile di doversi arrabattare per corrompere persone. credete forse che la risposta sia semplicemente aumentare la sicurezza fino ad arrivare a usare documenti di identità ufficiali? bene, se volete accaparrarvene qualcuno, potete iniziare a dare un'occhiata qui, ma tenete presente che ci sono organizzazioni criminali specializzate che sono molto più avanti di voi e, anche se tutte quelle sotterranee sono state chiuse, i governi ostili creeranno sicuramente milioni di passaporti falsi se siamo abbastanza stupidi da realizzare sistemi che rendono redditizio questo tipo di attività. e questo senza nemmeno menzionare gli attacchi nella direzione opposta, le istituzioni emittenti di identità che cercano di privare di potere le comunità marginali negando loro i documenti di identità... collusione. dato che così tanti meccanismi sembrano fallire allo stesso modo quando entrano in causa più identità o anche mercati liquidi, ci si potrebbe chiedere: c'è qualche elemento comune che causa tutti questi problemi? direi che la risposta è affermativa, e l'"elemento comune" è questo: è molto più difficile, e probabilmente completamente impossibile, realizzare meccanismi che mantengano proprietà desiderabili in un modello in cui i partecipanti sono in grado di cospirare, piuttosto che in uno dove non possono farlo. la maggior parte di voi probabilmente ha già capito di cosa sto parlando; esempi specifici di questo principio sono alla base di norme consolidate, spesso leggi, che promuovono la concorrenza sul mercato e vietano cartelli che fissano i prezzi, la compravendita di voti e la corruzione. ma il problema è molto più profondo e più generalizzato. nella versione della teoria dei giochi che si concentra sulla scelta individuale (cioè la versione che presuppone che ogni partecipante prenda decisioni in modo indipendente e che non consenta la possibilità che gruppi di agenti agiscano come un'unica entità per il proprio vantaggio reciproco) ci sono prove matematiche che in ogni gioco deve esistere almeno un equilibrio stabile di nash e che chi crea i meccanismi abbia un'influenza molto forte sulla "progettazione" dei giochi al fine di ottenere risultati specifici. ma nella versione della teoria dei giochi che consente la possibilità che le coalizioni lavorino insieme, detta teoria dei giochi cooperativi, ** ci sono ** ampie classi di giochi** che non hanno esiti stabili da cui una coalizione non possa deviare in modo proficuo**. giochi di maggioranza, formalmente descritti come giochi di n agenti in cui qualsiasi sottoinsieme di più della metà di loro può ottenere una ricompensa fissa e dividerla al proprio interno; molte situazioni di governance aziendali, di politica e altri casi che si verificano nella vita di tutti i giorni fanno parte di questo set di giochi intrinsecamente instabili. in altre parole, se esiste una situazione con un pool fisso di risorse e un meccanismo stabilito per distribuire tali risorse, ed è inevitabilmente possibile per il 51% dei partecipanti cospirare per prendere il controllo delle risorse, indipendentemente dalla situazione attuale può sempre emergere una cospirazione che risulti redditizia per i partecipanti. tuttavia, tale cospirazione sarebbe a sua volta vulnerabile a nuove potenziali cospirazioni, ivi compresa forse una combinazione di cospiratori e vittime precedenti... e così via. turno a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 questo fatto, l'instabilità dei giochi di maggioranza secondo la teoria dei giochi cooperativi, è probabilmente molto sottovalutato come modello matematico generale semplificato del perché potrebbe non esserci una "fine della storia" in politica e un sistema che si riveli pienamente soddisfacente. personalmente credo che sia molto più utile del più famoso teorema di arrow , per esempio. esistono due modi per aggirare questo problema. il primo è cercare di limitarci alla classe di giochi che siano realmente "privi di identità" e "a prova di collusione", dove non dobbiamo preoccuparci né di tangenti né di identità. il secondo è tentare di attaccare direttamente i problemi legati all'identità e alla collusione, e risolverli abbastanza bene da poter implementare giochi non a prova di collusione, con le maggiori proprietà che questa condizione può offrire. progettazione di giochi privi di identità e a prova di collusione la classe di giochi privi di identità e a prova di collusione è considerevole. anche il proof of work è a prova di collusione fino al limite di un singolo attore con ~23,21% degli hash totali, e questo limite può essere aumentato fino al 50% con una progettazione intelligente</1 >. i mercati competitivi sono ragionevolmente a prova di collusione fino a un limite relativamente alto, che è facilmente raggiungibile in alcuni casi, ma non in altri. nel caso della governance e della selezione di contenuti (i quali sono entrambi solo casi speciali del problema generale, dato dalla capacità di identificare buoni e cattivi), un'importante classe di meccanismi che funziona bene è la futarchia, in genere definita come "governance in base al mercato della previsione", sebbene direi che anche l'uso di depositi cauzionali si situi fondamentalmente nella stessa classe di tecniche. il modo in cui i meccanismi di futarchia, nella loro forma più generale, lavorano è dato dal fatto che rendono il "votare" non solo un'espressione di opinione, ma anche una previsione, con una ricompensa per le previsioni che si rivelano vere e una sanzione per quelle che si rivelano sbagliate. ad esempio, la mia proposta di "mercati previsionali per il dao di selezione di contenuti" suggerisce una progettazione semi-centralizzata, dove chiunque può votare positivamente o negativamente i contenuti inviati, dove i contenuti con più voti positivi sono più visibili, e dove è presente anche un "comitato moderatore" che prende le decisioni finali. per ogni post, c'è una piccola probabilità (proporzionale al volume totale di voti positivi e negativi a quel post) che il comitato moderatore sia chiamato a prendere una decisione definitiva sul post. se il comitato moderatore approva un post, tutti coloro che lo hanno votato positivamente sono ricompensati, mentre chi ha votato negativamente viene penalizzato; se il comitato moderatore disapprova un post accade il contrario. questo meccanismo incoraggia i partecipanti a dare voti positivi e negativi con l'intento di "prevedere" il giudizio del comitato moderatore. un altro possibile esempio di futarchia è un sistema di governance per un progetto con un token, dove chi vota a favore di una decisione sia obbligato ad acquistare una certa quantità di token al prezzo fissato al momento dell'inizio della votazione se il voto risulta vincente; questo fa sì che un voto per una decisione sbagliata sia costoso, e nel limite, se una cattiva decisione ottiene un voto, chiunque abbia approvato la decisione deve essenzialmente rilevare tutti gli altri partecipanti al progetto. questo garantisce che un singolo voto per una decisione "sbagliata" possa essere molto costoso per chi l'ha espresso, precludendo la possibilità di attacchi di corruzione a basso costo. una descrizione grafica di una forma di futarchia, dove vengono creati due mercati che rappresentano i due "possibili mondi futuri" e viene scelto quello con un prezzo più favorevole. fonte* questo post su ethresear.ch tuttavia, vi è un limite a ciò che questi meccanismi possono fare. nel caso dell'esempio di selezione dei contenuti appena descritto, non stiamo realmente risolvendo la governance, stiamo solo scalando la funzionalità di un gadget di governance che si presume sia già attendibile. si potrebbe cercare di sostituire il comitato moderatore con un mercato di previsione sul prezzo di un token che rappresenta il diritto di acquistare spazio pubblicitario, ma nella realtà i prezzi sono un indicatore troppo forte per rendere tale processo attuabile per un numero molto limitato di decisioni di ampia portata. e spesso il valore che cerchiamo di massimizzare è esplicitamente qualcosa di diverso dal valore massimo di una moneta. diamo un'occhiata più esplicita sul perché, in un caso più generale in cui non sia facile determinare il valore di una decisione di governance attraverso il suo impatto sul prezzo di un token, i buoni meccanismi per l'identificazione del bene e del male pubblico non possono purtroppo essere senza identità o a prova di collusione. se si cerca di preservare la capacità di un gioco di essere privo di identità costruendo un sistema in cui le identità non contano e solo le monete contano, si verifica un compromesso impossibile tra l'incapacità di incentivare i beni pubblici legittimi e l'appoggio eccessivo della plutocrazia. l'argomentazione è la seguente. supponiamo che un autore stia producendo un bene pubblico (ad esempio un blog con una serie di post) che fornisce valore a ogni membro di una community di 10000 persone. supponiamo che esista un meccanismo per cui i membri della community possono intraprendere un'azione che fa sì che l'autore riceva un guadagno di $1. a meno che i membri della comunità siano estremamente altruistici, perché il meccanismo funzioni, il costo di questa azione deve essere di gran lunga inferiore a $1, altrimenti la parte del beneficio ottenuta dal membro della community che sostiene l'autore sarebbe molto inferiore al costo richiesto per sostenere l'autore, e così il sistema diventerebbe una tragedia dei luoghi comuni in cui nessuno supporta l'autore. pertanto, deve esistere un modo per far guadagnare all'autore $1 a un costo di gran lunga inferiore a $1. ma ora supponiamo che esista anche una community fasulla, composta da 10000 account fake di un solo facoltoso hacker. questa community svolge esattamente le stesse azioni di quella reale, ma anziché sostenere l'autore, sostiene un altro account fake, anch'esso di proprietà dell'hacker. se fosse possibile per un membro della "vera community" dare all'autore $1 a un costo personale molto inferiore a $1, sarebbe possibile anche per l'hacker dare a se stesso $1 a un costo molto inferiore a $1 per tutte le volte che desidera, finendo per esaurire i fondi del sistema. qualsiasi meccanismo che possa aiutare parti veramente poco coordinate a coordinarsi senza le giuste tutele aiuterà anche le parti già coordinate (ad esempio molti account controllati dalla stessa persona) a coordinarsi ancora di più, sottraendo denaro al sistema. la problematica è analoga quando l'obiettivo non è finanziare, ma stabilire quale contenuto deve essere più visibile. quale contenuto pensate possa ottenere più valore in dollari: un articolo di blog di qualità legittimamente alta utile a migliaia di persone, ma la cui utilità per ogni singolo individuo è relativamente scarsa, o questo? o forse questo? chi ha seguito i recenti avvenimenti di politica "nel mondo reale" potrebbe anche segnalare un altro tipo di contenuto, che va a vantaggio di attori altamente centralizzati: la manipolazione dei social media da parte di governi ostili. in ultima analisi, sia i sistemi centralizzati che quelli decentrati si trovano ad affrontare lo stesso problema di fondo, cioè che il "mercato delle idee" (e dei beni pubblici più in generale) è molto lontano dall'essere un "mercato efficiente" con il significato normalmente attribuito al termine dagli economisti, e questo porta sia a una produzione limitata di beni pubblici anche in "tempo di pace", che alla vulnerabilità ad attacchi attivi. è veramente un problema grave. è anche il motivo per cui sistemi di voto basati su moneta (come bihu) hanno un grande vantaggio rispetto ai sistemi basati sull'identità (come gitcoin clr o l'esperimento con donut di /r/ethtrader): almeno non c'è interesse ad acquistare account in massa perché tutto quello che si fa è proporzionale al numero di monete che si possiede, a prescindere dal numero di account sui quali queste sono suddivise. tuttavia, meccanismi che non si basino su alcun modello di identità e che si affidino esclusivamente alle monete non possono risolvere il problema degli interessi concentrati che surclassano le piccole community che cercano di sostenere i beni pubblici; un meccanismo privo di identità che dà poteri alle comunità distribuite non può evitare di assegnare potere eccessivo ai plutocrati centralizzati che si fingono comunità distribuite. i beni pubblici però non sono solo vulnerabili ai problemi di identità, ma anche alla corruzione. per capire perché, considerate nuovamente l'esempio qui sopra, dove invece della "community fasulla" con 10001 account fake dell'hacker, l'aggressore ha solo un'identità, l'account che riceve finanziamenti, e gli altri 10000 account sono utenti reali, ma utenti che ricevono una tangente di $0,1 ciascuno per eseguire l'azione che farebbe guadagnare all'hacker un ulteriore $1. come già accennato, queste tangenti possono essere incredibilmente nascoste, anche attraverso servizi di custodia di terzi che votano per conto di un utente in cambio di convenienza; nel caso di strutture con "voto tramite moneta" è ancora più facile nascondere le tangenti: si possono noleggiare monete sul mercato e utilizzarle per partecipare al voto. così, mentre alcuni tipi di giochi, in particolare il mercato delle previsioni o i giochi basati su deposito cauzionale, possono essere realizzati a prova di collusione e privi di identità, il finanziamento generalizzato di beni pubblici sembra essere una categoria di problema in cui purtroppo non è possibile adottare approcci a prova di collusione e privi di identità. resistenza alla collusione e identità l'alternativa consiste nel prendere il problema di petto. come già detto, passare semplicemente a sistemi di identità centralizzati di sicurezza superiore, come passaporti e altri documenti di identità ufficiali, non funzionerà su larga scala; in un contesto sufficientemente incentivato sono molto insicuri e vulnerabili agli stessi governi che li emettono. in realtà, il tipo di "identità" di cui parliamo qui è una sorta di gruppo di attestazioni, robusto e a più fattori, il cui scopo è dimostrare che un attore identificato da un gruppo di messaggi è effettivamente una persona unica. un proto-modello molto precoce di questo tipo di identità in rete è discutibilmente il recupero sociale di chiavi nei telefoni blockchain di htc: l'idea di base è che la vostra chiave privata sia suddivisa in segreto tra un massimo di cinque contatti affidabili, in modo tale da garantire in modo matematico che tre di loro siano in grado di recuperare la chiave originale, ma almeno due non possano. si tratta di un "sistema di identità": sono i vostri cinque amici che determineranno se chi sta cercando di recuperare il vostro account siete proprio voi. si tratta però di un sistema di identità a scopi speciali, che cerca di risolvere un problema (la sicurezza degli account personali) diverso (e più semplice) dal tentativo di identificare esseri umani unici. detto questo, il modello generale che prevede individui che fanno dichiarazioni su altri individui può essere ampliato in una sorta di modello di identità più robusto. questi sistemi potrebbero essere aumentati, se si desidera, utilizzando il meccanismo di "futarchia" descritto in precedenza: se qualcuno afferma che un altro è un essere umano unico, qualcun altro non è d'accordo ed entrambe le parti sono disposte a impegnarsi per dirimere la controversia, il sistema può convocare una commissione di giudici che decida chi ha ragione. ma vogliamo anche un'altra condizione di fondamentale importanza: vogliamo un'identità che non si possa credibilmente noleggiare o vendere. ovviamente, non possiamo impedire alle persone di fare un accordo "tu mi invii $50", io ti mando la mia chiave", ma quello che possiamo cercare di fare è evitare che tali accordi siano credibili, in modo che il venditore non possa ingannare facilmente l'acquirente e inviargli una chiave che non funziona. un modo per raggiungere questo risultato è creare un meccanismo con il quale il proprietario possa inviare una transazione che revochi la chiave e la sostituisca con un'altra chiave di sua scelta, in un modo che non possa essere dimostrato. forse il modo più semplice per raggiungere l'obiettivo è usare una parte fidata che esegua il calcolo e pubblichi solo i risultati (insieme a prove di conoscenza zero che dimostrino i risultati, così la parte fidata è affidabile solo per la privacy, non per l'integrità) o decentralizzare la stessa funzionalità attraverso un calcolo a più parti. questo tipo di approccio non risolverà completamente il problema della collusione (un gruppo di amici potrebbe ancora riunirsi e coordinare i voti) ma lo ridurrà almeno in misura gestibile per non portare al completo fallimento di questi sistemi. c'è anche un altro problema: la distribuzione iniziale della chiave. cosa succede se un utente crea la propria identità all'interno di un servizio di custodia di terzi che poi archivia la chiave privata e la usa per votare in modo clandestino? si tratterebbe di corruzione implicita, il potere di voto dell'utente in cambio di un servizio comodo per l'utente, e per di più, se il sistema è sicuro in quanto riesce ad evitare la corruzione rendendo i voti non provabili, anche il voto clandestino da parte di host terzi non sarebbe rilevabile. l'unico approccio per aggirare il problema sembrerebbe essere.... la verifica di persona. ad esempio, si potrebbe creare un ecosistema di "emittenti" in cui ogni emittente rilasci smart card con chiavi private, che l'utente può scaricare immediatamente sul proprio smartphone; l'utente potrebbe poi inviare un messaggio per sostituire la chiave con una diversa che non rivelerà a nessuno. questi emittenti potrebbero essere conoscenti o potenzialmente individui già considerati affidabili tramite determinati meccanismi di voto. realizzare l'infrastruttura per rendere possibili meccanismi a prova di collusione, che includano solidi sistemi di identità decentrati, è una sfida difficile, ma se vogliamo sbloccare il potenziale di tali meccanismi, sembra inevitabile che dobbiamo fare del nostro meglio per provare. è vero che l'attuale dogma della sicurezza informatica, ad esempio per introdurre il voto online, è semplicemente "non si fa", ma se vogliamo espandere il ruolo dei meccanismi di voto, comprese forme più avanzate come il voto quadratico e la finanza quadratica, ad un maggior numero di ruoli, non abbiamo altra scelta se non affrontare di petto la sfida, tentare e ritentare, sperando di riuscire a realizzare qualcosa di abbastanza sicuro, almeno per alcuni casi di utilizzo. dark mode toggle über kollusion 2000 jan 01 see all posts besonderer dank geht an glen weyl, phil daian und jinglan wang für das gegenlesen dieses artikels, und an inlak16 für die übersetzung. in den letzten jahren hat sich das interesse erhöht, absichtlich entwickelte wirtschaftliche anreize und mechanismus-design zu nutzen, um das verhalten der teilnehmer in verschiedenen zusammenhängen in einklang zu bringen. im raum der blockchain bietet das mechanismus-design in erster linie die sicherheit für die blockchain selbst, indem miner oder proof-of-stake-validatoren zu ehrlicher teilnahme ermutigt werden. in jüngerer zeit jedoch wird es in vorhersagemärkten, "token-kuratierte register" und vielen anderen zusammenhängen angewandt. die aufkeimende radicalxchange-bewegung hat inzwischen experimente mit harberger-steuern, quadratischer abstimmung, quadratischer finanzierung und weiteres hervorgebracht. in jüngerer zeit wächst auch das interesse an der nutzung von token-basierten anreizen, um hochwertige beiträge in den sozialen medien zu fördern. da sich die entwicklung dieser systeme von der theorie zur praxis annähert, gibt es eine reihe von herausforderungen, die angegangen werden müssen. herausforderungen, denen wir meines erachtens noch nicht ausreichend begegnet sind. das jüngstes beispiel für diesen schritt von der theorie in richtung einsatz ist bihu, eine chinesische plattform, die kürzlich einen tokenbasierten mechanismus veröffentlicht hat, der leute zum schreiben von beiträgen motiviert. der grundlegende mechanismus (siehe whitepaper in chinesisch hier) ist, dass, wenn ein nutzer der plattform key-token hält, haben sie die möglichkeit, diese key-token auf artikel zu setzen. jeder benutzer kann k "befürwortungen" pro tag setzen, und das "gewicht" jeder positiven bewertung ist proportional zum einsatz des benutzers, der die stimmabgabe vornimmt. artikel mit einer größeren menge an sie aufwertenden einsätzen (stakes) erscheinen prominenter und der autor eines artikels erhält eine belohnung in form von key-tokens in etwa proportional zur menge an key-token, die den artikel "befürwortet" haben. das ist eine übertriebene vereinfachung und der eigentliche mechanismus beinhaltet einige nichtlinearitäten. aber diese sind nicht wesentlich für das grundlegende funktionieren des mechanismus. der key hat wert, da er auf verschiedene arten innerhalb der plattform verwendet werden kann. vor allem aber wird ein prozentsatz aller werbeeinnahmen verwendet, um key zu kaufen und zu verbrennen (yay, große daumen hoch für sie, dass sie nicht noch einen weiteren tauschmittel-token erschaffen haben !). dieses design ist alles andere als einzigartig. das motivieren zur erstellung von online-inhalten ist etwas, worum sich sehr viele menschen bemühen. es gibt viele designs mit ähnlichem charakter, sowie einige ziemlich unterschiedliche designs und in diesem fall wird diese spezielle plattform bereits signifikant genutzt: vor ein paar monaten hat das ethereum trading-subreddit /r/ethtrader eine teilweise ähnliche experimentelle funktion eingeführt, in der ein token namens "donuts" an benutzer ausgegeben wird, die positiv bewertete kommentare verfasst haben. eine bestimmte wöchentliche menge von donuts wird an die nutzer proportional zu der anzahl der positiven bewertungen, die ihre beiträge bekommen haben, ausgegeben. die donuts können verwendet werden, um das recht zu kaufen, den inhalt des banners am kopf des subreddits zu bestimmen, sowie bei meinungsumfragen zu wählen. im gegensatz zu den vorgängen im key-system ist hier die belohnung die b erhält, wenn a b positiv bewertet, nicht proportional zur vorhandenen token-menge von a. stattdessen hat jedes reddit-konto die gleiche fähigkeit, zu anderen reddit-konten beizutragen. diese art von experimente, bei denen die erstellung qualitativ hochwertiger inhalte in einer weise belohnt wird, die über die bekannten grenzen von spenden/kleinst-trinkgelder hinausgeht, sind sehr wertvoll. die unzureichende kompensation von benutzergenerierten internetinhalten ist in der gesellschaft im allgemeinen ein sehr erhebliches problem (siehe "liberal radicalism" und "data as labor") und es ist ermutigend zu sehen, wie krypto-communities versuchen, die macht des mechanismus-designs zu nutzen, um bei der lösung voranzukommen. aber leider sind diese systeme auch anfällig für angriffe. /0> selbstabstimmung, plutokratie und bestechungsgelder.** hier kann man das oben vorgeschlagene design wirtschaftlich angreifen. angenommen, ein wohlhabender benutzer erwirbt eine menge n von token, und als ergebnis jeder der k upvotes des benutzers gibt dem empfänger eine belohnung von n * q (q hier wahrscheinlich eine sehr kleine zahl, z.b. denke q = 0.000001). der benutzer befürwortet einfach mit stimmen seine eigenen pseudo-accounts und gibt sich die belohnung von n * k * q letztendlich selbst. dann fällt das system einfach zusammen, indem jeder benutzer einen "zinssatz" von k * q pro periode erhält und der mechanismus nichts anderes erreicht. der eigentliche bihu-mechanismus scheint dies vorherzusehen und hat eine überlineare logik, in der artikel mit mehr befürwortenden key-tokens eine unverhältnismäßig höhere belohnung erhalten, um die befürwortung populärer artikel zu ermutigen, anstatt selbst-promotion zu fördern. es ist ein gängiges muster unter den abstimmungssystemen für tokens, diese art von superlinearität hinzuzufügen, um zu verhindern, dass die selbst-promotion das gesamte system untergräbt. die meisten dpos-schemen haben eine begrenzte anzahl von delegierten slots mit null-belohnungen für alle, die nicht genug stimmen erhalten, um einem der slots beizutreten mit ähnlicher wirkung. doch führen diese regelungen unweigerlich zu zwei neuen schwächen: sie subventionieren plutokratie, da sehr wohlhabende menschen und kartelle immer noch genügend mittel zur selbst-promotion bekommen können. sie können umgangen werden, indem nutzer andere nutzer bestechen, um en masse für sie zu stimmen. bestechungsangriffe können weit hergeholt klingen (wer hat hier jemals eine bestechung im wirklichen leben akzeptiert?), aber in einem ausgereiften ökosystem sind sie viel realistischer, als es scheint. in den meisten zusammenhängen, in denen bestechung im blockchain-space stattgefunden hat, verwenden die betreiber einen euphemistischen neuen namen, um dem konzept ein freundliches gesicht zu geben: es ist keine bestechung, es ist ein "staking-pool", der "dividendenden teilt". bestechungsgelder können sogar verschleiert werden: stellen sie sich eine kryptowährung vor, die ein system ohne gebühren anbietet und die anstrengungen auf sich nimmt, eine ungewöhnlich gute benutzeroberfläche auf die beine zu stellen, und zudem nicht einmal versucht, gewinne zu sammeln. stattdessen verwendet sie token, die von den benutzern hinterlegt werden, um an verschiedenen tokenwahlsystemen teilzunehmen. es wird unweigerlich auch leute geben, die absprachen innerhalb von gruppen als ganz normal ansehen; sieh dir zum beispiel den jüngsten skandal um eos dpos an: schließlich gibt es die möglichkeit einer "negativen bestechung", dh. erpressung oder nötigung. in diesem fall wird den teilnehmern mit schaden gedroht, es sei denn, sie handeln innerhalb des mechanismus auf eine bestimmte weise. im /r/ethtrader-experiment hat angst vor menschen, die hinzukommen, um donuts zu kaufen und abstimmungen zu verfälschen, dazu geführt, dass die gemeinschaft sich dazu entschlossen hat, nur gesperrte (dh. unhandelbare) donuts für die stimmabgabe zu verwenden. aber es gibt einen noch günstigeren angriff als den kauf von donuts (ein angriff, den man als eine art verschleierte bestechung bezeichnen kann): sie zu verleihen . wenn ein angreifer bereits eth hält, kann er es als sicherheiten auf einer plattform wie compound verwenden, um eine anleihe mit irgendeinem token aufzunehmen. gleichzeitig gibt es ihm das volle recht, diesen token für welchen zweck auch immer, einschließlich der teilnahme an abstimmungen zu verwenden. wenn er fertig ist, schickt er einfach den token an den kreditvertrag zurück, um die sicherheiten zurück zu bekommen all das ohne auch nur eine sekunde der preisexposition gegenüber dem token, mit dem er gerade eine tokenabstimmung manipuliert hat, ausgesetzt gewesen zu sein, selbst wenn der tokenwahlmechanismus eine zeitsperre enthält (wie zb. bihu). in jedem fall sind probleme rund um bestechung und versehentlich übermäßige ermächtigung gut vernetzter und wohlhabender teilnehmer überraschend schwer zu vermeiden. identität einige systeme versuchen, die plutokratischen aspekte der tokenabstimmung durch verwendung eines identitätssystems zu mildern. im fall des /r/ethtrader-donut-systems zum beispiel wird, obwohl die governance-umfragen über tokenabstimmung durchgeführt werden, der mechanismus zur bestimmung der donut ausschüttung (dh. token), an reddit-konten gebunden: 1 positive stimme von 1 reddit-konto = verdient n donuts. das ideale ziel eines identitätssystems ist, den erhalt einer identität zu vereinfachen, aber den erhalt vieler identitäten zu erschweren. im /r/ethtrader-donut-system sind das reddit-konten, im gitcoin clr matching gadget sind es github konten, die für den gleichen zweck verwendet werden. aber identität, zumindest wie sie bisher eingesetzt wurde, ist eine anfällige angelegenheit.... oh, bist du zu faul um eine große menge an telefonen zu nutzen? vielleicht suchst du hiernach: eine gewöhnliche warnung darüber, wie dubiose seiten dich betrügen können oder auch nicht, mache deine eigene recherche usw. trifft hier zu. vermutlich ist das angreifen dieser mechanismen, indem man einfach tausende gefälschte identitäten wie ein puppenmeister kontrolliert noch einfacher, als menschen zu bestechen. und wenn du denkst, dass die antwort nur darin besteht, die sicherheit auf das niveau von regierungsebenen zu erhöhen? nun, wenn du ein paar davon erhalten möchtest, kannst du dich hier erkundigen, aber bedenke, dass es spezialisierte kriminelle organisationen gibt, die dir weit voraus sind. selbst wenn alle kriminellen strukturen zerschlagen würden, gibt es feindselige regierungen, die definitiv gefälschte pässe in millionenanzahl erstellen werden, wenn wir so dumm sind, systeme zu schaffen, die diese art von aktivität rentabel machen. und dies bezieht nicht einmal angriffe in die entgegengesetzte richtung mit ein, wenn identitätsausgabeinstitutionen versuchen, marginalisierte gemeinschaften durch das leugnen ihrer identitätsdokumente zu entmachten... kollusion angesichts der tatsache, dass so viele mechanismen auf ähnliche weise zu scheitern scheinen, wenn mehrere identitäten oder sogar liquide märkte ins bild kommen, könnte man sich fragen, ob es einen tief verwurtelzen, gemeinsamen grund gibt, der all diese probleme verursacht? ich würde argumentieren, die antwort lautet ja; und der "gemeinsame grund" lautet folgendermaßen: es ist viel schwieriger und wahrscheinlich unmöglich, mechanismen zu erschaffen, die wünschenswerte eigenschaften in einem modell erhalten, in dem die teilnehmer zusammenarbeiten können, als in einem modell, wo sie es nicht können. die meisten leute haben wahrscheinlich schon eine gewisse intuition in dieser hinsicht. spezifische beispiele dieses prinzips liegen hinter etablierten normen und oft hinter gesetzen zur förderung wettbewerbsfähiger märkte und zur einschränkung von preisfestlegungen, kauf und verkauf von stimmen und bestechung. aber die problematik geht viel tiefer und in die breite. in der version der spieltheorie, die sich auf die individuelle wahl konzentriert (das heißt, die version, die davon ausgeht, dass jeder teilnehmer selbständig entscheidet und die nicht die möglichkeit bietet, dass gruppen von agenten zum gegenseitigen nutzen zusammen arbeiten können), gibt es mathematische beweise, dass mindestens ein stabiles nash-gleichgewicht in jedem spiel existieren muss. mechanismus-designer haben hier einen sehr breiten spielraum, spiele zu "konstruieren", um bestimmte ergebnisse zu erzielen. aber in der version der spieltheorie, die es ermöglicht, in koalitionen zusammenzuarbeiten, genannt kooperative spieltheorie, gibt es große klassen von spielen , die kein stabiles ergebnis haben, von dem eine koalition nicht rentabel abweichen kann. mehrheitsspiele werden formell als spiele von n akteuren beschrieben, bei denen jede teilmenge von mehr als der hälfte von ihnen eine feste belohnung erlangen und unter sich aufteilen kann. ein setup, das vielen situationen in der unternehmensführung, politik und vielen anderen situationen im menschlichen leben unheimlich ähnlich ist, sind teil dieser menge von inhärent instabilen spielen. das heißt, wenn es eine situation mit einem festen pool von ressourcen und einem mechanismus für die verteilung dieser ressourcen gibt und es unvermeidlich möglich ist, dass sich 51% der teilnehmer verschwören können, um die kontrolle über die ressourcen zu übernehmen, egal, was die aktuelle konfiguration ist, gibt es immer eine verschwörung, die entstehen kann, die für die teilnehmer profitabel wäre. allerdings wäre diese verschwörung dann wiederum anfällig für potenzielle neue verschwörungen, möglicherweise auch für eine kombination früherer verschwörer und opfer. und so weiter und so weiter. runde a b c 1 1/3 1/3 1/3 2 1/2 1/2 0 3 2/3 0 1/3 4 0 1/3 2/3 diese tatsache, die instabilität von mehrheitsspielen im rahmen der kooperativen spieltheorie, wird wohl als vereinfachtes mathematisches modell massiv unterschätzt, weswegen es in der politik kein "ende der geschichte" und kein system gibt, das sich als vollkommen zufriedenstellend erweist. ich persönlich glaube, dass es viel nützlicher ist, als beispielsweise das berühmtere arrow-theorem. es gibt zwei wege, dieses problem zu umgehen. die erste besteht darin, uns auf die klasse von spielen zu beschränken, die "identitätsfrei" und "kollusionssicher" sind, also wo wir uns nicht um bestechungen oder identitäten sorgen machen müssen. die zweite besteht darin, die probleme der identitätsund kollusionsresistenz direkt anzugehen und sie tatsächlich gut genug zu lösen, um absprachefreie spiele mit den besseren eigenschaften, die sie bieten, zu implementieren. identitätsfreies und absprachefreies spieldesign die klasse von spielen, die identitätsfrei und absprachesicher sind, ist beträchtlich. sogar proof of work ist absprachesicher bis hin zu einem einzigen akteur mit ~23,21% der gesamten hashpower. diese bindung kann mit cleverer konstruktion auf bis zu 50% erhöht werden. wettbewerbsfähige märkte sind einigermaßen absprachesicher bis zu einer relativ hohen grenze, die in einigen fällen leicht erreicht werden kann, in anderen fällen jedoch nicht. im falle von governance und inhaltspflege (beides sind wirklich nur sonderfälle des allgemeinen problems der identifizierung öffentlichen gemeinwohls und öffentlichen nachteils) ist eine gut funktionierende klasse von mechanismen futarchy typischerweise als "führung nach vorhersagemärkten" dargestellt, obwohl ich auch argumentieren würde, dass der einsatz von sicherheitseinlagen grundsätzlich in der gleichen klasse der technik liegt. in ihrer allgemeinsten form funktionieren futarchy-mechanismen, indem sie nicht nur zum ausdruck der meinung "abstimmen", sondern auch eine vorhersage mit einer belohnung für wahre vorhersagen und einer strafe für falsche vorhersagen machen. beispielsweise schlägt mein vorschlag für "vorhersagemärkte für inhaltspflegende daos" ein semi-zentralisiertes design vor, bei dem jeder eingereichte inhalte befürworten oder ablehnen kann. positiv bewertete inhalte werden stärker sichtbar und ein "moderation panel" trifft die endgültigen entscheidungen. für jeden beitrag gibt es eine geringe wahrscheinlichkeit (proportional zum gesamtvolumen von upvotes+downvotes auf diesem beitrag), dass das moderations-panel aufgerufen wird, um eine endgültige entscheidung über den beitrag zu treffen. wenn das moderations-panel einen beitrag billigt, wird jeder, der ihn gewählt hat, belohnt und jeder, der ihn abgelehnt hat, bestraft. und wenn das moderations-panel einen beitrag ablehnt, findet das rückwärtsverfahren anwendung. dieser mechanismus ermutigt die teilnehmer, befürwortungen und ablehnungen zu vergeben, die versuchen, die bewertungen des moderationspersonals "vorherzusehen". ein weiteres mögliches beispiel für futarchy ist ein verwaltungssystem für ein projekt mit einem token in dem jeder, der für eine entscheidung stimmt, verpflichtet ist, zum zeitpunkt der abstimmung eine gewisse menge an tokens zum aktuellen preis zu kaufen, wenn die abstimmung gewinnt. dadurch wird sichergestellt, dass die abstimmung für eine schlechte entscheidung kostspielig ist und im extremfall, wenn eine schlechte entscheidung eine abstimmung gewinnt, müssen alle, die die entscheidung gebilligt haben, im wesentlichen alle anderen in dem projekt herauskaufen. dies stellt sicher, dass eine individuelle abstimmung über eine "falsche" entscheidung für die wähler sehr kostspielig sein kann und die möglichkeit billiger bestechungsangriffe ausschließt. eine grafische beschreibung einer form der futarchy, schaffung von zwei märkten, die die beiden "möglichen zukunftswelten" repräsentieren und die diejenige mit dem günstigeren preis wählen. quelle dieser beitrag auf ethresear.ch die bandbreite von dingen, die mechanismen dieser art leisten können, ist jedoch begrenzt. im fall des inhaltspflegebeispiels lösen wir nicht wirklich governance, wir skalieren nur die funktionalität eines governance-gadgets, das bereits als vertrauenswürdig angenommen wurde. man könnte versuchen, das moderations-panel durch einen vorhersagemarkt zu ersetzen, der das recht auf den kauf von werbeflächen repräsentiert. aber in der praxis sind die preise zu volatil als indikator, um dies für alles andere als eine sehr geringe zahl von sehr großen entscheidungen zu ermöglichen. und oft ist der wert, den wir zu maximieren versuchen, explizit etwas anderes als der maximalen wert eines tokens. schauen wir uns genauer an, warum im allgemeineren fall, in dem wir den wert einer governance-entscheidung nicht einfach durch ihre auswirkung auf den preis eines tokens bestimmen können, gute mechanismen zur identifizierung öffentlichen gemeinwohls und öffentlichen nachteils leider nicht identitätsfrei oder absprachefrei sein können. wenn man versucht, die eigenschaften eines spiels identitätsfrei zu halten, in dem ein system gebaut wird, wo identitäten keine rolle spielen und nur tokens entscheidend sind, gibt es einen unmöglichen kompromiss zwischen dem versäumnis, legitimes öffentliches gemeinwohl anzuregen oder der übermäßigen subventionierung der plutokratie. das argument lautet wie folgt: nehmen wir an, es gibt einen autor, der ein öffentliches gut produziert (zb. eine reihe von blog-beiträgen), das wert für jedes mitglied einer gemeinschaft von 10000 personen bietet. angenommen, es gibt einen mechanismus, bei dem mitglieder der gemeinschaft eine aktion durchführen können, die dem autor einen gewinn von $1 zukommen lässt. sofern die community-mitglieder nicht extrem altruistisch sind, müssen, damit der mechanismus funktioniert, die kosten für die durchführung dieser aktion viel niedriger sein als $1, da sonst der anteil des nutzens, den das mitglied der gemeinschaft für den autor einnimmt, viel geringer wäre als die kosten für die unterstützung des autors. und so bricht das system in eine tragödie des allgemeinguts zusammen, wo niemand den autor unterstützt. daher muss es eine möglichkeit geben, den autor $1 verdienen zu lassen, zu kosten von weit weniger als $1. aber jetzt nehmen wir an, dass es auch eine gefälschte gemeinschaft gibt, die aus 10000 gefälschten pseudo-konten desselben wohlhabenden angreifers besteht. diese community führt alle gleichen aktionen wie die wirkliche gemeinschaft durch, außer den autor zu unterstützen. sie unterstützen ein anderes gefälschtes konto, das auch ein pseudo-account des angreifers ist. wenn es einem mitglied der "echten gemeinschaft" möglich ist, dem autor $1 zu persönlichen kosten von weit weniger als $1 zu geben, ist es für den angreifer möglich, sich selbst immer wieder $1 zu einem preis von weniger als $1 zu geben und damit dem system die finanzierung zu entziehen. jeder mechanismus, der wirklich wenig koordinierten parteien bei der koordinierung helfen kann wird, ohne die richtigen schutzmechanismen auch bereits koordinierten parteien helfen (wie viele konten, die von derselben person kontrolliert werden) zuviel zu koordinieren, wodurch geld aus dem system entnommen wird. eine ähnliche herausforderung stellt sich dann, wenn das ziel nicht die finanzierung ist, sondern zu bestimmen, welche inhalte am sichtbarsten sein sollen. welche inhalte würden deiner meinung nach durch mehr dollarwert unterstützung finden: ein legitimer hochwertiger blog-artikel, der tausenden von menschen zugute kommt, aber jedem einzelnen relativ wenig zugute kommt, oder dies? oder vielleicht dies? diejenigen, die die jüngste politik "in der realen welt" verfolgt haben, könnten auch auf eine andere art von inhalten hinweisen, von denen hoch zentralisierte akteure profitieren: die manipulation der sozialen medien durch feindliche regierungen. letzten endes stehen sowohl die zentralisierten systeme als auch die dezentralen systeme vor dem gleichen grundlegenden problem. der "marktplatz der ideen" (und allgemein öffentlicher güter) ist sehr weit von einem "effizienten markt" entfernt in dem sinne, dass ökonomen normalerweise den begriff verwenden und dies führt sowohl zur unterproduktion öffentlicher güter selbst in "friedenszeit" und auch zur anfälligkeit für aktive angriffe. es ist nur ein schwieriges problem. dies ist auch der grund, warum tokenbasierte abstimmungssysteme (wie bihu) einen großen echten vorteil gegenüber identitätsbasierten systemen haben (wie dem gitcoin clr oder dem /r/ethtrader donut experiment): zumindest gibt es keinen nutzen für den kauf von konten en masse. denn alles, was du tust, ist proportional zu der anzahl der token, die du hast, unabhängig davon, auf wie viele konten die münzen aufgeteilt sind. allerdings können mechanismen, die sich nicht auf irgendein identitätsmodell stützen und sich nur auf token stützen, das problem konzentrierter interessen, die gegen zersplitterte das gemeinwohl unterstützende gemeinschaften die oberhand haben, nicht grundsätzlich lösen. ein identitätsfreier mechanismus, der verteilte gemeinschaften ermächtigt, kann es nicht vermeiden, zentralisierte plutokraten zu ermächtigen, die vorgeben, verteilte gemeinschaften zu sein. aber es sind nicht nur identitätsfragen, die spiele öffentlichen gemeinwohls angreifbar machen, sondern auch bestechungen. um das zu erkennen, solltest du dir das obige beispiel erneut anschauen, aber anstelle der "fake community" mit 10001 pseudo-accounts des angreifers, hat der angreifer nur eine identität, das geld erhaltende konto und die anderen 10000 konten sind echte benutzer aber benutzer, die eine bestechung von je $0,01 für die handlung erhalten, die dem angreifer den erhalt von zusätzlich $1 verursacht. wie bereits erwähnt, können diese bestechungsgelder stark verschleiert werden, auch durch die dienste von dritten, die gegen bequemlichkeit im namen eines benutzers abstimmen und im falle der tokenabstimmung ist ein verschleiertes bestechungsgeld sogar noch einfacher: man kann token auf dem markt mieten und sie für die teilnahme an den abstimmungen nutzen. einige arten von spielen, vor allem vorhersagemärkte oder sicherheits-einzahlungs-basierte spiele, können kollusionssicher und identitätsfrei gemacht werden. die allgemeine finanzierung öffentlichen gemeinguts hingegen scheint eine problemklasse zu sein, in der kollusionssichere und identitätsfreie ansätze leider nicht funktionieren. kollusionsresistenz und identität die andere alternative ist, das identitätsproblem direkt anzugreifen. wie bereits erwähnt, werden die zentralisierten identitätsysteme mit höherem sicherheitsniveau, wie pässe und andere staatliche ausweise nicht in großem maßstab funktionieren. in einem ausreichend begünstigten kontext sind sie sehr unsicher und anfällig gegenüber den ausstellenden regierungen selbst! die art von "identität", über die wir hier sprechen, ist eher eine art robuste, multifaktorielle reihe von forderungen, dass ein akteur, der durch eine reihe von nachrichten identifiziert wird, tatsächlich ein einzigartiges individuum ist. ein sehr frühes modell dieser art von vernetzter identität ist wohl "soziale erholung" in htc's blockchain smart-phone: die grundidee ist, dass dein privater schlüssel zwischen bis zu fünf vertrauenswürdigen kontakten geteilt wird, so dass mathematisch sichergestellt ist, dass drei von ihnen den ursprünglichen schlüssel zurückholen können, aber zwei oder weniger nicht. dies qualifiziert sich als "identitätssystem" es sind deine fünf freunde, die entscheiden, ob jemand versucht, dein konto wiederherzustellen oder nicht. allerdings ist es ein spezielles identitätssystem, das versucht, ein problem zu lösen persönliche kontosicherheit das sich von dem problem des versuchs, einzigartige menschen zu identifizieren, unterscheidet (und einfacher ist!). allerdings kann das allgemeine modell von einzelpersonen, die untereinander ansprüche erheben, durchaus in eine art robusteres identitätsmodell eingebunden werden. diese systeme könnten auf wunsch mit der oben beschriebenen "futarchy"-mechanik erweitert werden: wenn jemand behauptet, jemand sei ein einzigartiger mensch, jemand anderes anderer meinung ist und beide seiten bereit sind, in eine anleihe einzuzahlen, um das problem zu klären, kann das system ein urteilsgremium zusammenrufen, um zu bestimmen, wer recht hat. aber wir wollen auch ein anderes wichtiges eigenschaft: wir wollen eine identität, die man nicht glaubwürdig mieten oder verkaufen kann. natürlich können wir die leute nicht daran hindern, ein geschäft zu machen "schicke mir $50 und ich sende dir meinen schlüssel", aber wir können versuchen, zu verhindern, dass solche geschäfte glaubwürdig wirken sodass der verkäufer den käufer leicht betrügen und dem käufer einen schlüssel geben kann, der nicht wirklich funktioniert. eine möglichkeit ist, einen mechanismus zu schaffen, durch den der eigentümer eines schlüssels eine transaktion senden kann, die den schlüssel widerruft, ihn durch einen anderen schlüssel der wahl des eigentümers ersetzt und das alles in einer weise, die nicht bewiesen werden kann. vielleicht ist der einfachste weg, dies zu umgehen, entweder eine vertrauenswürdige partei zu verwenden, die die berechnung betreibt und nur ergebnisse veröffentlicht (zusammen mit zero knowledge beweisen, die die ergebnisse belegen, damit der vertrauenswürdigen partei nur aus gründen der privatsphäre, nicht aus gründen der integrität vertraut wird) oder die gleiche funktionalität durch mehrparteien-berechnung dezentralisiert. derartige ansätze werden das problem der kollusion nicht vollständig lösen. eine gruppe von freunden könnte zusammenkommen und auf der gleichen couch sitzen und abstimmungen koordinieren, aber sie würden es zumindest auf ein überschaubares ausmaß reduzieren, das nicht zu einem völligen versagen dieser systeme führen wird. es gibt noch ein weiteres problem: die anfängliche verteilung des schlüssels. was passiert, wenn ein nutzer seine identität in einem drittanbieter-gewahrsam erstellt, der dann den privaten schlüssel speichert und ihn dazu verwendet, heimlich über dinge abstimmen zu lassen? dies wäre eine implizite bestechung, die stimmrechte des benutzers im gegenzug für die bereitstellung eines bequemen dienstes einfordert. und wenn das system insofern sicher ist, als es bestechungsgelder erfolgreich verhindert, dass stimmen nicht nachweisbar sind, wäre die geheime stimmabgabe durch dritte auch nicht nachweisbar. der einzige ansatz, der dieses problem umgeht: personenüberprüfung. zum beispiel könnte man ein ökosystem von "herausgebern" haben, in dem jeder herausgeber chipkarten mit privaten schlüsseln ausgibt, die der benutzer sofort auf sein smartphone herunterladen und eine nachricht senden kann, um den schlüssel durch einen anderen schlüssel zu ersetzen, den er niemandem preisgibt. diese herausgeber könnten treffen und konferenzen sein, oder potenziell personen, die von einigen wahlmechanismen bereits als vertrauenswürdig eingestuft wurden. der aufbau der infrastruktur, um kollusionsresistente mechanismen wie robuste dezentrale identitätsysteme möglich zu machen ist eine schwierige herausforderung. aber wenn wir das potenzial solcher mechanismen wirklich freischalten wollen, scheint es unvermeidlich, unser bestes zu tun, um es zu ausprobieren. es stimmt, dass zum beispiel das derzeitige dogma für die computersicherheit, zum beispiel für die einführung der online-abstimmung einfach "mach es nicht" ist. aber wenn wir die rolle der abstimmungsmechanismen, einschließlich fortgeschrittener formen, wie quadratische abstimmungen und quadratische finanzen, auf mehr rollen erweitern wollen, haben wir keine andere wahl, als der herausforderung frontal zu begegnen. wir müssen wirklich hart daran arbeiten und hoffentlich gelingt es, zumindest für einige anwendungsfälle, mechanismen zu finden, die sicher genug sind. ideologies and imaginaries in blockchain communities: the case of ethereum ideologies and imaginaries in blockchain communities: the case of ethereum ann brody mcgill university stéphane couture université de montréal abstract background: academic literature on blockchains has focused on bitcoin, which is traditionally associated with right-wing libertarianism. this article looks at ethereum, an alternative that emerged in canada and is now the second most used blockchain technology after bitcoin. analysis: using participatory observation supplemented with publicly available material, this article examines the ideologies and imaginaries surrounding ethereum and how they are articulated with its technical design. conclusion and implications: ethereum’s design ostensibly widens the ideological spectrum of cryptocurrency while “masking” certain currency ideologies still prominent within it. this complicates the distinction seen in the literature between blockchain as currency and blockchain as media and points to the increasing need to study non-currency-based blockchain technologies. keywords: blockchain; ethereum; ideologies; decentralization; science and technology studies résumé contexte : la recherche sur les blockchains s’est surtout attardé à bitcoin, en l’associant aux idéologies libertariennes. cet article aborde ethereum, la technologie de blockchain la plus utilisée après bitcoin. analyse : basé sur l’observation participante et du matériel publiquement accessible, l’article analyse les idéologies et imaginaires entourant ethereum et leur articulation avec son design technique. conclusion et implications : ethereum élargit le spectre idéologique des blockchains tout en «masquant» certaines idéologies monétaires toujours proéminentes. cela complique la distinction énoncée dans la littérature entre blockchains comme monnaie et blockchains comme média, et souligne la nécessité d’étudier davantage les usages nonmonétaires des blockchains. brody, ann, & couture, stéphane. ideologies and imaginaries in blockchain communities: the case of ethereum. canadian journal of communication 46(3), 543–561. doi:10.22230/cjc.2021v46n3a3701 ©2021 ann brody & stéphane couture. cc by-nc-nd h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 https://doi.org/10.22230/cjc.2021v46n3a3701 544 canadian journal of communication, vol 46(3) mots clés : blockchain; ethereum; idéologies; décentralisation; études sur les sciences et les technologies introduction this article examines the ideologies and imaginaries of ethereum—the second most popular blockchain and cryptocurrency technology after bitcoin. blockchains are often conflated with bitcoin and associated with cyberlibertarianism (golumbia, 2016; scott, 2014). however, a growing number of studies and analyses have shed light on the diversity of blockchain-based technologies and the values associated with them (dupont 2019; husain, 2020; scott, 2016; swan, 2015). while it is true that ethereum, bitcoin, and many other decentralized currency projects all rely on blockchain technologies, they also significantly vary in their goals, visions, and imaginaries. for instance, while the bitcoin (n.d.) website presents it as “an innovative payment network and a new kind of money” (para. 1), ethereum (n.d.) describes itself as “the world’s programmable blockchain” (para. 2). this article will analyze this difference in visions and values alongside the design and discourses surrounding ethereum and its social imaginaries. ethereum is worth studying within a canadian context as it was created in the early 2010s by vitalik buterin, a russian-canadian programmer who spent his early life in toronto. at a 2013 bitcoin meet up in toronto, he shared a white paper describing ethereum as a kind of protocol that promised a vast number of usage cases. notably, from its earliest origins, ethereum was envisioned as a global platform for enacting smart contracts and building decentralized applications. while the canadian dimension of this technology is not the focal point of this article, it is relevant to canadian communication scholarship. this article begins by presenting a brief literature review on blockchain technologies. it will outline the analytical and methodological framework used herein and then move on to describe ethereum and analyze some of its core values and imaginaries. it will highlight some of the aspects of ethereum that resemble and deviate from bitcoin. specifically, the “world computer” (antonopoulos & wood, 2018, “introduction to ethereum”, para. 1) is invoked to characterize the ideological underpinnings of ethereum as blockchain as the new infrastructure of the internet for any kind of usage case (i.e., beyond its currency aspect). in particular, this imaginary of the world computer leads to situating ethereum in line with hacker cultures and ideologies, especially the ethos of tinkering (coleman, 2012) and the idea of a recursive public, defined by christopher kelty (2008) as a public that is vitally concerned with the means of its own existence. still, looking closer at ethereum’s design, there is a persistent prevalence of the currency feature and its influence on the ideological landscape of the project. this article’s main argument is that ethereum straddles the distinction between blockchain 1.0 and blockchain 2.0 and challenges a clean divide between them (swan, 2015; scott, 2016). while ethereum h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca widens the spectrum of uses offered by bitcoin, it still retains certain decisive design features of bitcoin. two main design features of ethereum—the cryptocurrency and decentralized applications—are analyzed in relationship with different imaginaries, with a focus on how these features reflect and reinforce the cyberlibertarian ideologies and imaginaries of bitcoin that continue to persist in ethereum, despite being ideologically “masked” (flichy, 2007, p. 99) to some extent by the world computer imaginary. ethereum has certainly been addressed in the literature as part of the broader category of blockchain technologies and has been the object of specific inquiries (bracamonte & okada, 2017; dupont, 2017a; fairley, 2019; fenu, marchesi, marchesi, & tonelli, 2018; gerard, 2017), but there is a need for a more detailed case study of ethereum, specifically one that probes its social imaginaries. following the argument that imaginaries have influenced the development of the internet (flichy, 2007), examining ethereum’s imaginaries should help illuminate what the new computing paradigm that ethereum is actively building toward might mean— or not mean—for our sociotechnical futures. this analysis is especially important as ethereum is now the second most important blockchain and is sometimes considered typical of the use of blockchain beyond cryptocurrency. the objective here is to provide a case study that specifically examines the mutual shaping of ethereum’s ideologies, imaginaries, and design, and to advocate more broadly for the study of these interrelations in other blockchain projects that are ostensibly established for purposes beyond market transaction. ideologies and imaginaries of blockchains: a brief literature review in recent years, there has been a proliferation of studies surrounding blockchains and cryptocurrencies from a multitude of disciplines and perspectives. for instance, the blockchain research network, edited by quinn dupont, listed at the time of writing more than 3,000 papers related to the subject, including about 1,300 in the “social and human” category. within this literature, a growing set of studies focus on the ideologies and imaginaries of blockchain and cryptocurrency communities, a subject this article aims to contribute to. within these works, bitcoin has attracted the most attention (brunton, 2019; de filippi & loveluck, 2016; dodd, 2017; dupont, 2014; golumbia, 2016; karlstrøm, 2014; maurer, nelms, & swartz, 2013; redshaw, 2017; scott, 2014; vidan & lehdonvirta, 2019), partly due to its comparatively longer existence. two works are particularly relevant to this article. first, an early and polemical work by david golumbia (2016) called the politics of bitcoin: software as rightwing extremism. here, the author argues that blockchain technologies, and bitcoin in particular, are premised on right-wing politics, both in their design and ideological constitution. providing a more nuanced perspective, lana swartz (2018) identified two imaginaries that shaped the development of bitcoin: digital metallism, referring to the belief that money supply should be determined by internabrody & couture the case of ethereum 545 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 tional currency markets rather than state policies (and should be free from state control), and infrastructural mutualism, which views cryptocurrencies as a decentralized platform for moving money, leading to more freedom of information and creating the potential for an alternative to centralized banking. swartz goes on to explain how these two distinct economic imaginaries influenced bitcoin and led to the rise of ideological contentions within the bitcoin community. other authors have looked beyond bitcoin to study the potentialities of blockchains more broadly in a shift that some have attributed as a move from blockchain 1.0 to blockchain 2.0 (scott, 2016; swan, 2015). for melanie swan (2015), blockchain 1.0 is primarily concerned with cryptocurrencies whereas blockchain 2.0 relates more to “contracts.”1 referring to blockchain-based protocols such as ethereum, counterparty, and omni, aaron wright and primavera de fillipi (2015) note that blockchain technology enables not only decentralized currencies but also “self-executing digital contracts (smart contracts) and intelligent assets that can be controlled over the internet (smart property)” (p. 1). similarly, brett scott (2016) also sets bitcoin apart from other blockchain 2.0 technologies and projects such as ethereum, counterparty, and blockstream, which he contends have “more overtly communitarian ideals” (p. ii) and “potential for creating cooperation at scale” (p. ii). in particular, scott (2016) identifies two visions of blockchains: the “conservative libertarian” view grounded in a criticism of statecontrolled currencies, echoing golumbia’s perspective, and the “communitarian anarchist” vision advocating for “non-hierarchical and solidarity-based” (p. 15) systems. dupont (2019) also revisits some common narratives found in crypto and blockchain communities and envisions cryptocurrencies and blockchains as two distinct media forms. he argues that the shift from bitcoin—which is seen as money—to blockchain technologies in the general sense has resulted in “an abstraction (and elimination) of ideology and meaning” (p. 43), as they are now less about money and more about “purer tools” (p. 43) due to the greater possibilities they offer as open ledger systems. it is thus important to critically examine how these divergent value systems and visions of digital transactions map onto projects such as ethereum that do not neatly reflect this “purer” aim. analytical perspective and methodology golumbia (2016) and swartz’s (2018) analyses of the ideologies and imaginaries of bitcoin provide a starting point for this analysis. in fact, the first version of this article was intended as a counterargument to golumbia’s analysis of blockchains (in the case of bitcoin), which differed from how the authors understood ethereum blockchain at the time. while this study—through the literature review and empirical work—has developed a more complex view of blockchains and ethereum, golumbia’s work on bitcoin provides an interesting starting point that serves as a good comparative basis for this study of ethereum. dupont (2017), who 546 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca also criticizes golumbia for his polemical and one-sided approach, notably refers to a specific aspect of golumbia’s work: golumbia also makes a case that these political commitments are integrated into the design of the technical system, for example, through seemingly innocuous choices to limit algorithmically the supply of bitcoins (ostensibly) to curb inflation, or by requiring computational “proof of work” to create new coins (as a metaphor for gold mining). (p. 474) following this approach, this analysis aims to look at the articulation of political views within the design of the technical system of ethereum. further, it takes more broadly into consideration “imaginaries,” drawing on swartz’s (2018) analysis of distinct imaginaries within bitcoin (digital metallism and infrastructural mutualism) to enrich the understanding of ethereum. the analytical goal is thus to look at the ideologies and imaginaries of ethereum and how they are articulated in its design as a technical system. this approach is situated within scholarly work in science and technology studies (sts) (akrich, 1992; flichy 2008; jasanoff & kim, 2015) as well as in ethnographical work on software communities (coleman, 2012; kelty, 2008), looking at the articulation of values, imaginaries, and technological design. for instance, in sts, and within actor-network theory in particular, madeleine akrich (1992) writes that much of the work of technological design involves “inscribing” a certain vision of the world into the technical content of the new object, which, in turn, contains a “script” (p. 208) or a scenario of the way it should be used. contemporary sts works also look at this articulation of design through the concept of sociotechnical imaginaries, which sheila jasanoff and sang-hyun kim (2015) define as “collectively imagined forms of social life and social order reflected in the design” (p. 4). patrice flichy (2007) also proposes a model for the role of “technical imaginaries,” (p. 13) relying on paul ricoeur’s distinction between utopias and ideologies, where utopias are used to open possibilities while ideologies are used to legitimize new technology projects. in particular, flichy (2007) uses the concept of “mask ideology” (p. 11) to refer to a utopia that has become an ideology, partly masking a reality but also mobilizing actors. common to all these works is a sociotechnical perspective that sees technological design and worldviews (taking the form of values, imaginaries, ideologies) as mutually shaping and constituting each other. this article also draws on the ethnographies of software communities (often intersecting with sts perspectives) to analyze how ideologies play out in software design. christopher kelty (2008) and gabriella coleman (2012), for instance, note the limitations and difficulties that classical ideological frameworks (such as “right wing,” “left wing,” “socialist,” or “liberal”) present when studying software communities. coleman (2012) remarks on the “political agnosticism” that characterbrody & couture the case of ethereum 547 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 izes free and open source software developers who “actively disavow political associations that go beyond software freedom,” permitting them to “escape the various ideological polarizations” (p. 22). kelty (2008) writes that ideologies, in relation to free software, cannot be apprehended a priori but rather emerge from technological practices: “geeks do not start with ideologies, but instead come to them through their involvement in the practices of creating free software and its derivatives” (p. 8). many of the descriptions for this case study are informed by ann brody’s active participation in the ethereum community during the last three years, including one full year as an ma student (brody, 2019), during which she conducted formal participatory observation at ethereum’s fourth annual conference, devcon iv, in 2018, and conducted seven interviews, some of which were used in this article2. brody also attended other ethereum international events and local meet ups, and these observations are supplemented in this article by references and the analysis of materials that are “publicly available”—to use kelty’s (2008, p. 21) term—and produced by ethereum actors. these materials were not collected or analyzed in any systematic fashion but rather from an ethnographically inspired perspective.3 the case of ethereum ethereum was introduced by vitalik buterin, a russian-canadian programmer who spent his early life in toronto, canada. buterin reportedly got involved in bitcoin as early as 2011 (peck, 2016), and he soon started imagining a protocol that could enable greater use cases. at a 2013 bitcoin meet up in toronto, buterin shared a white paper describing his vision of ethereum as a blockchain-based platform that promised vast possibilities beyond monetary transactions (filiba, 2018). that document, a next-generation smart contract & decentralized application platform, described how blockchains could be used to enable applications such as smart contracts and what buterin (2013) has referred to as “decentralized applications” (p. 11). similar to bitcoin, ethereum is based on blockchain, a technology that at its core involves creating a chain of records or “blocks” that are cryptographically linked to each other. theoretically, it is impossible to retroactively modify one of the blocks without altering the whole chain. as with bitcoin, ethereum’s blockchain is also distributed, meaning that the blockchain is collectively managed on a peer-to-peer network instead of a central organization. for this reason, a distributed blockchain is often compared to a public digital accounting ledger that makes it possible to record and store data in a transparent and immutable way.4 while the concept of blockchain was first implemented in bitcoin to support cryptocurrencies, it is now used for much broader goals. this is where ethereum differs from bitcoin. as the title of buterin’s (2013) white paper states, a defining feature of ethereum is its ability to execute smart contracts, which are computer 548 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca programs intended to automatically execute a transaction on the blockchain according to predefined terms. while some features of smart contracts are implemented in bitcoin to exchange currency, they can be used in ethereum to exchange anything that holds value (such as property or shares) without requiring the services of third parties. another defining feature of ethereum, and central to the argument here, is its ability to facilitate decentralized applications, usually referred to as dapps. these applications function in a similar way to apple ios or android apps but are decentralized and run on the blockchain instead of on centralized servers. for example, cryptokitties—a blockchain game that allows players to buy, sell, and exchange digitally animated cats—was once one of the most popular dapps on ethereum. by being validated through the blockchain, each cryptokitty is unique and cannot be transferred (bought or sold) without the agreement of its owner. status is another popular dapp that combines a peer-to-peer messenger, a wallet (for storing cryptocurrencies), and a secure web browser into a private and secure communication tool. in his white paper, buterin (2013) envisioned many kinds of dapps use cases, such as financial derivatives, identity systems, decentralized file storage, and even “decentralized autonomous organizations” (para. 3). despite this defining difference, ethereum still shares many similarities with bitcoin. ethereum has its native cryptocurrency, called ether, that can be traded on the market in the same manner as bitcoin. for instance, on april 16, 2020, one ether (or eth) was worth cad$240.30. in 2020, there were approximately 110.5 million ether in circulation, for a total worth of about cad$26 billion (malwa, 2020). this cryptocurrency is also used for paying fees for running smart contracts, dapps, or any other feature requiring transactions on the ethereum blockchain. moreover, as with bitcoin, creating a new currency token in ethereum is called mining—in reference to gold mining—and involves a complex computational process called proof of work that consists of confirming new, recently made transactions and adding a new “block.” this mining process often involves vast amounts of computer resources and has been criticized for its energy requirements and ecological costs (egiyi & ofoegbu, 2020). for this reason, ethereum is currently developing ethereum 2.0 to replace proof of work with proof of stake, a process that reduces mining resources and is thus more ecologically sustainable. at the time of writing, ethereum 2.0 was being run in test mode. the following sections analyze three themes prevalent in ethereum to show how it distinguishes itself from some of bitcoin’s imaginaries while also reproducing others. the first section addresses the imaginary of the “world computer.” the second describes the culture of experimentation and the ethos of building evident in ethereum. the third section shows how, despite these described imaginaries and values that appear to be different from bitcoin, discourses on cryptocurrency and cyberlibertarianism still persist in ethereum. after presenting these empirical brody & couture the case of ethereum 549 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 analyses, the last section discusses whether or not ethereum can be characterized as “right wing” by bringing it into conversation with golumbia’s (2016) analysis of bitcoin. beyond currency: the imaginary of the “world computer” ethereum promises a vast range of uses of the blockchain that extend beyond cryptocurrency transactions. in terms of design, ethereum can be understood as a “turing-complete” virtual machine, meaning that any kind of computational problem could theoretically be solved using ethereum, as its script is not limited to solely facilitating monetary transactions. in particular, ethereum’s smart contracts are usually written in a programming language called solidity that is also geared toward turing-complete operation. this is unlike bitcoin’s programming language, known as script, which is limited to executing simple currency transactions. in his white paper, buterin (2013) explains the open potential of ethereum’s turing-completeness: what ethereum intends to provide is a blockchain with a built-in fully fledged turing-complete programming language that can be used to create “contracts” that can be used to encode arbitrary state transition functions, allowing users to create any of the systems described … , as well as many others … yet imagined, simply by writing up the logic in a few lines of code. (p. 1) akrich’s (1992) theorization of scripts can be used here to emphasize the distinction between ethereum’s flexible design and bitcoin’s more restricted design. ethereum’s design as a virtual machine, and especially its programming language solidity, is scripted in a way that imposes minimal user constraints, promoting different courses of actions and preferences. this promotes a broader ideological orientation than what is permitted by bitcoin’s limited programming language, which is primarily intended for financial transactions. indeed, this design characteristic of ethereum has been a source of many imaginative futures aligning with the “world computer” imaginary. this term seems to move through ethereum’s history (cuen, 2019; gupta, 2005; quénetain, 2018; wu, 2020). it was coined at the first devcon conference in 2015 by ethereum cofounder gavin wood, who gave a presentation called “ethereum: the world’s computer.” buterin spoke of ethereum in these terms when he described it as a platform that would enable developers to execute their code in exchange for a fee: [ethereum is] a massively replicated turing-complete state machine, to which anyone in the world with the ability to buy 0.3 cents worth of ether can upload code which every participant in the network will be forced to run on their local machine. (ogundeji, 2016, para. 13) in short, this perspective imagines ethereum as a type of computer that does not reside in a single system but is ubiquitous. applications are run on a vastly decen550 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca tralized computer system that cannot ever be shut down and that anyone with access to an internet connection can implement. other similar terms could be associated with the world computer imaginary. for instance, a developer at devcon iv described how he envisions ethereum’s prospective use cases: distribution of power and wealth in a more egalitarian way. that would be the ideal impact, although simply the use of this technology in the backbone of the internet (for example, dns) will make the internet and digital communication more secure and guaranteed compared to what it is today. (alex, interview 3) while this quote hints at blockchain’s broader political aims, the idea of ethereum being the new “internet backbone” imagines ethereum restructuring internet architecture, allowing people that do not necessarily know or trust each other to settle transactions. it also emphasizes a kind of internet model where monopolized power cannot take hold. a related term put forward by ethereum co-founder gavin wood is web 3.0, or simply web3, the idea of a new version of the web where information that is exchanged is authenticated or “trusted” through the blockchain. wood (2018) described his aspiration in the following way: web 3.0 will engender a new global digital economy, creating new business models and markets to go with them, busting platform monopolies like google and facebook, and giving rise to vast levels of bottom-up innovation. cheap government attacks on our privacy and liberty like widespread data trawling, censorship, and propaganda, will become more difficult. (para. 19) this new web model could empower users to be in charge of their data thanks to blockchain’s encryption techniques that, in theory, would enable users to maintain pseudo-anonymity and share personal information only on a permissioned basis. events such as account suspensions, denials of services, and service disruptions would also be significantly reduced. in certain ways, this perspective is in continuity with many historical or contemporary peer-to-peer or decentralized network projects, such as the interplanetary file system, free net, and mojo nation. these networks implement distributed data storage, meaning that when one downloads files, one is downloading them from someone else’s computer rather than a central server. unlike these file-sharing projects, ethereum can be used to automate broader transactions in an authenticated way (through the feature of smart contracts), thus pursuing the original idea of the world computer, instead of just a file system.5 in some respects, the world computer imaginary aligns with swartz’s (2018) characterization of mutual infrastructuralism in the case of ethereum, which is brody & couture the case of ethereum 551 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 imagined as a new decentralized and mutualized computing infrastructure. however, swartz’s imaginaries are primarily concerned with economics and monetization. the world computer imaginary, on the contrary, encompasses a vision of blockchain, the internet, and even computing that goes well beyond economic usages. and yet, rather than a rupture, the imaginary of the world computer can probably be best characterized as an extension of mutual infrastructuralism to encompass a broader vision of infrastructures. at the same time, it is important to note that digital metallism—swartz’s other imaginary, which emphasizes the provision of an alternative to state money—still seems to be present in ethereum; namely, it continues to prevail through the metaphor of “mining,” inspired by gold standard economics. buidl versus hodl: the ethos of building and experimentation in ethereum during an interview, an ethereum developer named moe made an interesting comparison between the ethereum community, which he described as inclusive and experimental, and the bitcoin community, which he described as more “tribalistic”: i would describe ethereum as being more about openness, fairness, equality, and what i like about it is that you’re allowed to experiment more. … if you compare these kinds of values to bitcoin, they are more skeptical, more tribalistic, and they believe if you’re very open and nice to people, you’re going to open it up to all kinds of people that are going to change bitcoin. … people from different networks are welcome here because ethereum is all about experimentation. (moe, interview 7) certain ethereum developers said they were drawn to ethereum because of its intellectual challenge. one developer put it this way: i always wanted to learn coding, and hearing about blockchain finally became a reason to learn it and immerse myself into something that seemed to me completely unknown and a big challenge at the time. … i like that it’s open and inclusive. … cooperation instead of competition—people working on the same topic but don’t feel like competitors. (dan, interview 1) this mention of “cooperation instead of competition” is worth paying attention to, as it  suggests that ethereum’s commitments do not strictly fall under rigorous right-libertarian precepts of competition. two other developers expressed how ethereum’s culture played a significant part in sustaining their participation: ultimately [blockchain] keeps me engaged and brings me in contact with some of the most interesting, thoughtful, and kindest people i’ve ever met. (mark, interview 4) 552 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca ethereum has a strong underlying current of altruistic ideology. it influences our decision-making; it inspires our work and allows us to coordinate a multitude of projects that serve the greater good of the ethereum community. (aaron, interview 5) the goal is mainly to make widespread use of blockchain and streamline this technology throughout society. it’s open to new ideas, new people, and new use cases. i find the ethereum community really healthy and diverse. (alex, interview 3) in addition to the interviews we conducted at devcon iv, the conference demonstrated the sheer scope of possibilities for ethereum use, such as islamic finance, helping sex workers stay safe, and implementing tools of resistance in venezuela. this range of possibilities further magnifies the distinction between ethereum and bitcoin. indeed, different analysts, including those within blockchain communities, echo this distinction. for instance, kay kurokawa (2018), a software engineer active in crypto communities, sees bitcoiners placing a premium on restrictive access and securitization based on their rigorous cryptographic code of conduct. ethereum, on the other hand, is described as more inclusive and left leaning: “because their main instinct is to hope for best-case scenarios, their focus is on innovation … and innovation for left-wingers involves creating new ways to utilize the blockchain or inventing some alternative decentralized consensus” (para. 5). one of the discursive tropes prevalent in ethereum that captures this ethos of experimentation is the slang term buidl. at devcon iv, buidl appeared on customized conference stickers featuring vitalik buterin (see figure 1). an intentional misspelling of build, it is used to encourage the development of new blockchain applications to maintain the ethereum ecosystem. buidl is a sibling of the term hodl, which first appeared in a bitcoin forum in 2013 when a thread with an accidental misspelling of holding was posted by a user stating, “i am hodling” (bitcoin forum, 2013). hodl picked up cultural traction in the bitcoin community, encouraging crypto enthusiasts to hold on to their cryptocurrency rather than sell it and to stay strong when crypto prices dropped. buidl challenges hodl, to some extent, by promoting the building and expansion of the blockchain ecosystem rather than simply holding out for lucrative moments. brody & couture the case of ethereum 553 figure 1: conference sticker from devcon iv h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 the imaginary of the world computer and the ethos of building reveals how ethereum is committed to more than a market investment logic. indeed, the ethos of building resembles what coleman (2013) refers to as a kind of “tinkering” in her description of open source hacker communities. the world computer imaginary also points to ethereum as a recursive public, which has been defined by kelty (2008) in the case of free and open source software as a “public that is vitally concerned with the material and practical maintenance and modification of the technical, legal, practical, and conceptual means of its own existence as a public” (p. 3). it might be more apt to consider ethereum from the perspective of hacker ideologies which, as coleman (2013) and kelty (2008) note, are difficult to characterize on the left-right spectrum. ideologies, the authors write, emerge from technological making and are centred around the kinds of tinkering and experimenting associated with free and open source software and hacker communities. currency, finance, and the persistent prevalence of libertarianism while the imaginary of the world computer and the ethos of experimentation seem to be significatively present in ethereum and absent from bitcoin, ideological aspects related to currency that are normally associated with cyberlibertarianism nevertheless remain prevalent in ethereum. for example, ether, the native currency of ethereum, holds significant monetary value. it is also significant that vitalik buterin, for instance, has a net worth of over usd$21 billion, most of it relying on ether assets (chipolina, 2020). beyond the conversations that take place in the community concerning building, experimentation, and visions of the world computer, certain cyberlibertarian tendencies prevail in ethereum’s discourse. for instance, joe says: to change something, you have to build a new model that makes the existing model obsolete. if we design this system with that in mind, i see cryptocurrencies reducing the role of central parties that have been shown to mishandle the public treasury and financial institutions who reap the benefits of currency manipulation and inflation. (joe, interview 2) indeed, ethereum is well known for its use of initial coin offerings (icos) in creating other cryptocurrencies. functioning in the same manner as initial public offerings (ipos), which consist of offering shares of a company on the stock market, icos offer new kinds of tokens (or “coins”) that are expected to gain value in the future. it is worth mentioning that ethereum itself started as an ico, selling ethereum tokens (ether) that could be bought with bitcoin. in 2017, the total amount of funds raised by icos exceeded usd$4 billion (zetzsche, buckley, arner, & föhr, 2017), and as of february 2018, more than 1,000 icos had been held on ethereum, comprising 80 percent of all icos (fenu et al., 2018). icos were very quickly seen as “catnip for scammers” (morris, 2017, para. 3), given that they were 554 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca not initially regulated by any government bodies. it is not surprising that places such as china and south korea have outlawed them completely, and that they have gone through securities regulation in canada and the united states. they are evaluated in canada on a case-by-case basis to assess whether or not crypto is considered a “security” (for example, a tradable financial asset). another notion originating out of ethereum, which golumbia (2016) linked to cyberlibertarianism in his last chapter on the future of blockchain, is the decentralized autonomous organization (dao). a dao is conceptualized within blockchain communities as an organization (profit or nonprofit) whose constitutive rules and transactions are encoded within a blockchain. this helps shareholders and participants establish their contractual relationship without the need for central management or a central authority (as would be the case in a “classic” enterprise, which would need to register with a national government to be legally recognized). although daos are still in an experimental stage, the first one launched on ethereum played a significant role in ethereum’s early history. dubbed as “the mother of all daos” (hare, 2020, para. 2), it was created to serve as a business model for other daos and as a venture capital fund aiming to serve and collect funds for ethereum projects. it was not registered in any state and had no conventional management structure, such as a board of directors. instead, it operated according to rules that were encoded into the smart contracts that governed them. this “mother” dao is largely characterized as a failure that caused a significant schism in the community. in may 2016, when the dao crowdfund was launched, it earned its place as one of the biggest crowdfunds in history, amassing up to usd$120 million in digital currency (waters, 2016). a few weeks later, a hacker found a loophole in the code and stole around usd$50 million in ether from the fund. to repair the damage, the ethereum community decided to fork the code and create a new chain entirely. from the onset, many ethereum members were struck by this software fork. in the end, two ethereum blockchains stemmed from this event: ethereum classic (etc), the initial blockchain that was preserved based on “code is law,” and the more recent ethereum (eth), which is often associated with the idea of “extra-protocol intentionality,” meaning that it is the responsibility of the community to repair an injustice, even if this means changing the blockchain (de filippi, 2016). the latter blockchain is the focus of this article. a recent movement called defi (short for decentralized finance) has also had a marked impact on ethereum activity. defi, which is committed to shifting traditional financial products into decentralized ones that do not rely on government regulation or enterprises, is also a response to so-called centralized cryptocurrency exchanges that are owned by private businesses, especially as some of these have been the subject of allegations of fraud (kauflin, 2020). ethereum is a popular brody & couture the case of ethereum 555 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 blockchain for defi applications, partly due to its smart contract feature. one popular example is uniswap, a protocol that allows decentralized exchanges (or dex in the cryptocurrency vocabulary) between ether (eth) and other cryptocurrencies. defi pulse, a popular website for viewing the latest analytics and rankings of defi projects, reports that as of november 8, 2020, about usd$2.83 billion in crypto was “locked-in” to various defi projects.  defi has been compared to the 2017 ico craze, a period in crypto when prices started significantly rising overnight. with defi, however, more actors from conventional finance are participating (roberts, 2020). this renewed importance granted to finance points to the persisting prevalence of cyberlibertarianism, and a libertarian worldview more generally, in ethereum use. one actor who was interviewed for forbes magazine echoed this perspective, noting that many people are interested in defi because “they have a libertarian streak” (kauflin, 2020, para. 5), while others are simply fascinated by the technology itself. this points to a defining tension within ethereum between the aims of decentralizing finance and the more general ethos of building a new internet culture. discussion: between building and finance, is ethereum “right wing”? recall that golumbia (2016) contends that bitcoin and blockchain more generally are premised on right-wing values, both in their design and their ideological constitution. but what about ethereum? what are the politics of ethereum? a certain diversity of ideologies and imaginaries inform and sustain ethereum: while the significance of icos and the defi movement point to the influence of libertarian views on how ethereum is used, the importance granted to experimentation, openness, and inclusiveness seems to point to other ideological horizons that more closely align with the hacker ethics of tinkering (coleman, 2012) and the making and sharing of new and inclusive uses. this diversity of ideologies echoes some of the literature reviewed here, in particular what has been noted about the coexistence of communitarian and libertarian ideals within blockchain communities (dupont, 2019; scott, 2016). these conclusions also echo the findings reported by coindesk, a popular news site specializing in cryptocurrencies, which revealed in a survey that the majority of ethereum users (not necessarily the developers) identify as “liberals” and “socialists,” contrary to bitcoiners who identify more often as “conservative” or “libertarian” (ryan, 2018). it is safe to conclude that as a community, ethereum is best characterized as a space where different ideological horizons cohabitate while leaning decidedly toward the left. recall, however, that golumbia (2016) argues how bitcoin “in its very design” (p. 12) can only make sense within right-wing ideology. without necessarily adhering to golumbia’s right-wing imaginary, it has to be recognized that ethereum still shares many design features with bitcoin. the process of proof of work, associated 556 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 http://www.cjc-online.ca with gold mining, is one of these, which golumbia has criticized. more generally, ethereum, similar to bitcoin, has a native currency that plays a central role in successfully conducting transactions. other blockchain platforms, such as yosemite, have chosen to peg the value of their token on fiat government-approved money instead of relying on a native currency. for ethereum, rooting a native currency so centrally in its design is a choice, and an ideological one for that matter, that is echoed by the persistence of speculating practices and discourses. these are most notably expressed through the aforementioned ico craze and the new defi movement, which casts a light on the cyberlibertarian character of ethereum. that said, ethereum differs significantly from bitcoin based on the way that it supports turing-complete programming languages to enable (at least in theory) any kind of application to run on the blockchain. ethereum’s turing-completeness is antithetical to the kind of “right-wing” design that golumbia (2016) ascribes to blockchains in general.6 conclusion ethereum straddles what has been categorized in the literature as the distinction between blockchain 1.0 and 2.0 (scott, 2016; swan, 2015)—or, to use dupont’s (2019) characterization, between blockchain as money and blockchain as media more generally. while the stated purpose of ethereum is to be used in different ways beyond currency, defining itself according to the diverse potentials of blockchain 2.0, certain features of ethereum nevertheless remain grounded in finance and cryptocurrency culture. yet, rather than viewing the world computer imaginary and imaginaries related to finance and currency in opposition, it might be better to comprehend them as mutually reinforcing. in terms of design, this appears quite clearly in the way ether, as a currency, is used as a “gas” to fuel the ethereum as a world computer, which in turn reinforces the value and stability of the currency itself. in terms of ideology, the world computer ideal and related discourses of inclusive building and experimentation function as a “mask ideology” (flichy, 2007, p. 11). for flichy, utopian discourses concerning the revolutionizing impact of new technologies are later displaced or transformed in the cycle of technological development, minimizing or “masking” some other aspect of reality. the characterization of ethereum as a world computer is mobilized to enroll actors more inclined toward the building aspect of ethereum, while at the same time “masking” the fact that ethereum seems to be still foremost used as a financialization tool. future studies should examine more thoroughly the way ideologies and imaginaries not only co-exist or are in tension in blockchain communities but also how they might mask or legitimize each other. it would also be important to better locate or situate these ideologies and imaginaries within different categories of actors by looking at how they are differentially expressed by developers, investors, or orbrody & couture the case of ethereum 557 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 dinary users (brody, 2019). finally, and especially in the context of communication studies, more work should address the imaginaries of blockchain technologies that are not centred around currency and assess their potential for sustaining alternative digital infrastructures or acting as media (dupont, 2019). while libertarian worldviews seem to be prominent within cryptocurrency-based protocols, the situation might be quite different for blockchain projects intended to pursue other purposes, such as smart contracts in the case of ethereum. notes swan (2015) also mentions blockchain 3.0, which addresses “justice applications beyond currency, 1. economics, and markets” (p. 27). the first author is a white female who has experimented with decentral finance and markets on 2. ethereum for research purposes. the second author, a white male, was her first supervisor during her ma studies and has been active in research and advocacy work related free and open source software (couture, 2017, 2020). characterizing this approach as “ethnographically inspired” rather than firmly ethnographic draws 3. on hine’s approach (2000, 2007), which consists of maintaining a relationship with ethnography as a methodological “inspiration and source of guidance,” (hine, 2007, p. 668) without necessarily relying on canonical versions of what this methodology should be. for a more thorough description of blockchain, see dupont (2019). 4. some efforts are being made, however, to articulate blockchain and distributed file systems (see 5. huang, lin, zheng, zheng, & bian, 2020). ethereum distinguishes itself from bitcoin in other design details that are not addressed here. for 6. instance, while bitcoin has defined an algorithmic limit on the supply of bitcoin to curb inflation— something that golumbia (2016) associates with a right-wing perspective—there is no such fixed cap in ethereum, although this choice is sometimes called into question (sharma, 2019). ann brody is a doctoral student at mcgill university. email: ann.brody@mail.mcgill.ca. stéphane couture is assistant professor at université de montréal. email: stephane.couture @umontreal.ca. websites blockchain research network, www.blockchainresearchnetwork.org blockstream, https://blockstream.com coindesk, https://www.coindesk.com counterparty, https://counterparty.io defi pulse, https://defipulse.com free net, https://freenetproject.org interplanetary file system, https://ipfs.io mojo nation, https://mojo-nation.com omni, https://www.omnilayer.org yosemite, https://yosemitex.com references akrich, madeleine. (1992). the de-scription of technical objects. in w. bijker & j. law (eds.), shaping technology\building society: studies in sociotechnical (pp. 205–224). cambridge, ma: mit press. antonopoulos, andreas m., & wood, gavin. (2018). mastering ethereum: building smart contracts and dapps. url: https://cypherpunks-core.github.io/ethereumbook/01what-is.html [january 21, 2020]. 558 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 mailto:ann.brody@mail.mcgill.ca mailto:stephane.couture@umontreal.ca mailto:stephane.couture@umontreal.ca mailto:stephane.couture@umontreal.ca http://www.blockchainresearchnetwork.org https://blockstream.com https://www.coindesk.com https://counterparty.io https://defipulse.com https://freenetproject.org https://ipfs.io https://mojo-nation.com/ https://www.omnilayer.org https://yosemitex.com http://www.cjc-online.ca bitcoin. (n.d.). bitcoin [website]. url: https://bitcoin.org/en [january 2, 2020]. bitcoin forum. (2013, december 18). i am hodling [website]. url: https://bitcointalk.org/index.php ?topic=375643.0 [february 15, 2020]. bracamonte, vanessa, & okada, h. (2017, august). the issue of user trust in decentralized applications running on blockchain platforms. 2017 ieee international symposium on technology and society (istas), 2017, pp. 1-4, doi:10.1109/istas.2017.8318975 brody, ann. (2019). mapping ideologies in cryptocurrency and blockchain communities: the case of ethereum [unpublished m.a. thesis]. toronto, on: york university. brunton, finn. (2019). digital cash: the unknown history of the anarchists, utopians, and technologists who created cryptocurrency. princeton, nj: princeton university press. buterin, vitalik. (2013). ethereum white paper. github repository. url: https://github. com /ethereum/wiki/wiki/white-paper [september 8, 2020]. chipolina, scott. (2020). vitalik buterin sold half his bitcoin in 2013 to avoid going broke. decrypt. url: https://decrypt.co/51341/vitalik-buterin-sold-half-his-bitcoin-in-2013-to-avoid-going -broke [september 10, 2020]. coleman, gabriella. (2012). coding freedom: the ethics and aesthetics of hacking. princeton, nj: princeton university press. couture, stéphane. (2017). values and configuration of users in the design of software source code. international journal of communication, 11(2), pp. 1112–1132, http://ijoc.org/index.php /ijoc/article/view/4179 [november 1, 2020]. couture, stéphane. (2020). free and open source software. in m. o’neil, c. pentzold, & s. toupin (eds.), the handbook of peer production (pp. 153–168). hoboken, nj: wiley. doi:10.1002 /9781119537151.ch12 cuen, leigh. (2019, october 11). devcon shows ethereum’s “world computer” is a movement, not a product. coindesk. url: https://www.coindesk.com/devcon-shows-ethereums-world -computer-is-a-movement-not-a-product [november 5, 2020]. de filippi, primavera, & loveluck, benjamin. (2016). the invisible politics of bitcoin: governance crisis of a decentralised infrastructure. internet policy review, 5(3), 1–32. dupont, q. (2014). the politics of cryptography: bitcoin and the ordering machines.  journal of peer production,  1(4), 1–10. dupont, quinn. (2017a). experiments in algorithmic governance: a history and ethnography of “the dao,” a failed decentralized autonomous organization. in m. campbell-verduyn (ed.), bitcoin and beyond: cryptocurrencies, blockchains and global governance (pp. 157–178). new york, ny: routledge. dupont, quinn. (2017b). the politics of bitcoin: software as right-wing extremism, by david golumbia. journal of cultural economy, 10(5), 474–476. dupont, quinn. (2019). cryptocurrencies and blockchains. cambridge, uk: polity press. dodd, nigel. (2018). the social life of bitcoin. theory, culture & society,  35(3), 35–56. egiyi, modesta, & ofoegbu, grace. (2020). cryptocurrency and climate change: an overview. international journal of mechanical engineering and technology, 11(3), 15–22. ethereum. (n.d.). ethereum.org. url: https://ethereum.org/en/what-is-ethereum [january 2, 2020]. fairley, peter. (2018). ethereum will cut back its absurd energy use. ieee spectrum, 56(1), 29–32. fenu, gianni, marchesi, lodovica, marchesi, michele, & tonelli, roberto. (2018). the ico phenomenon and its relationships with ethereum smart contract environment. in 2018 international workshop on blockchain oriented software engineering, 26–32. ieee. filiba, jack. (2018). why ethereum co-founder vitalik buterin is a blockchain icon. coinsquare. url: https://news.coinsquare.com/blockchain/russian-canadian-ethereum-vitalik-buterin [may 11, 2020]. de filippi, primavera. (2016, july 11). a $50m hack tests the values of communities run by code. motherboard. url: https://www.vice.com/en_us/article/qkjz4x/thedao [february 24, 2020]. flichy, patrice. (2007). the internet imaginaire (1st ed.). cambridge, ma: mit press. gerard, david. (2017).  attack of the 50 foot blockchain: bitcoin, blockchain, ethereum & smart contracts. seattle, wa: createspace independent publishing platform. brody & couture the case of ethereum 559 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 https://bitcoin.org/en/ https://bitcointalk.org/index.php?topic=375643.0 https://bitcointalk.org/index.php?topic=375643.0 https://bitcointalk.org/index.php?topic=375643.0 https://doi.org/10.1109/istas.2017.8318975 https://decrypt.co/51341/vitalik-buterin-sold-half-his-bitcoin-in-2013-to-avoid-going-broke https://decrypt.co/51341/vitalik-buterin-sold-half-his-bitcoin-in-2013-to-avoid-going-broke https://decrypt.co/51341/vitalik-buterin-sold-half-his-bitcoin-in-2013-to-avoid-going-broke http://ijoc.org/index.php/ijoc/article/view/4179 http://ijoc.org/index.php/ijoc/article/view/4179 http://ijoc.org/index.php/ijoc/article/view/4179 https://doi.org/10.1002/9781119537151.ch12 https://doi.org/10.1002/9781119537151.ch12 https://doi.org/10.1002/9781119537151.ch12 https://www.coindesk.com/devcon-shows-ethereums-world-computer-is-a-movement-not-a-product https://www.coindesk.com/devcon-shows-ethereums-world-computer-is-a-movement-not-a-product https://www.coindesk.com/devcon-shows-ethereums-world-computer-is-a-movement-not-a-product https://ethereum.org/en/what-is-ethereum https://news.coinsquare.com/blockchain/russian-canadian-ethereum-vitalik-buterin https://www.vice.com/en_us/article/qkjz4x/thedao golumbia, david. (2016). the politics of bitcoin: software as right-wing extremism. minneapolis, mn: university of minnesota press. gupta, vinay. (2015, october 21). ethereum, smart contracts, and the world computer. consensys. url: https://consensys.net/blog/news/programmable-blockchains-in-context-ethereum -smart-contracts-and-the-world-computer/ [october 5, 2020]. hare, mercuryo. (2020, october 27). decentralized autonomous organization as a way to manage defi projects. mercuryo. url: https://medium.com/mercuryo/decentralized-autonomous -organization-as-a-way-to-manage-defi-projects-4905d4789e27 [november 12, 2020]. hine, c. (2000).  virtual ethnography. london: sage. hine, c. (2007). multi-sited ethnography as a middle range methodology for contemporary sts.  science, technology, & human values,  32(6), 652-671. huang, huawei, lin, jianru, zheng, baichuan, zheng, zibin, & bian, jing. (2020). when blockchain meets distributed file systems: an overview, challenges, and open issues. ieee access, 8(march 2020), 50574–50586. husain, syed omer, franklin, alex, & roep, dirk. (2020). the political imaginaries of blockchain projects: discerning the expressions of an emerging ecosystem. sustainability science, 15(march 2020), 379–394. jasanoff, sheila, & kim, sang-hyun. (2015). dreamscapes of modernity: sociotechnical imaginaries and the fabrication of power. chicago, il: university of chicago press. karlstrøm, henrik. (2014). do libertarians dream of electric coins? the material embeddedness of bitcoin.  distinktion: scandinavian journal of social theory,  15(1), 23–36. kauflin, jeff. (2020, december 16). why everyone in crypto is talking about defi. forbes. url: https://www.forbes.com/sites/jeffkauflin/2019/04/26/why-everyone-in-crypto-is-talking -about-defi/?sh=af68fe1723f3 [december 21, 2020]. kelty, christopher (2008). two bits the cultural significance of free software. durham, nc: duke university press. kurokawa, kay. (2018). left/right political ideologies of hodlers: part 1. hacker noon. url: https:// hackernoon.com/left-right-political-ideologies-of-hodlers-part-1-faa71ed689e4 [may 3, 2020]. malwa, shaurya. (2020, august 08). “how many ethereum in circulation?” asks the bitcoin community. cryptoslate. url: https://cryptoslate.com/how-many-ethereum-in-circulation-asks -the-bitcoin-community [november 12, 2020]. maurer, bill, nelms, taylor, & swartz, lana. (2013). “when perhaps the real problem is money itself!”: the practical materiality of bitcoin.  social semiotics,  23(2), 261–277. morris, david. (2017, may 31). the rise of cryptocurrency ponzi schemes. the atlantic. url: https:// www.theatlantic.com/technology/archive/2017/05/cryptocurrency-ponzi-schemes/528624 [may 13, 2020]. ogundeji, olusegun. (2016, june 8). vitalik buterin sets milestones on ethereum’s route to be the “world computer.” cointelegraph. url: https://cointelegraph.com/news/vitalik-buterin -sets-milestones-on-ethereums-route-to-be-the-world-computer [may 16, 2020]. peck, morgen (2016). the uncanny mind that built ethereum. backchannel. wired. url: https://www.wired.com/2016/06/the-uncanny-mind-that-built-ethereum [june 15, 2016]. quénetain, stanislas. (2018, march 13). ethereum: the first world computer. blockchains expert. url: https://www.blockchains-expert.com/en/ethereum-the-world-computer [october 5, 2019]. redshaw, tom. (2017). bitcoin beyond ambivalence: popular rationalization and feenberg’s technical politics.  thesis eleven,  138(1), 46–64. ryan, peter. (2018). left, right and center: crypto isn’t just for libertarians anymore. coindesk, url: https://coindesk.com/no-crypto-isnt-just-for-libertarians-anymore [may 10, 2019]. scott, brett. (2014). visions of a techno-leviathan: the politics of the bitcoin blockchain. e-international relations. url: https://www.e-ir.info/2014/06/01/visions-of-a-techno-leviathan-the-politics-of-the-bitcoin-blockchain [december 6, 2020]. scott, brett. (2016). how can cryptocurrency and blockchain technology play a role in building social and solidarity finance? (working paper no. 2016-1). geneva, ch: united nations research institute for social development. 560 canadian journal of communication, vol 46(3) h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 https://consensys.net/blog/news/programmable-blockchains-in-context-ethereum-smart-contracts-and-the-world-computer/ https://consensys.net/blog/news/programmable-blockchains-in-context-ethereum-smart-contracts-and-the-world-computer/ https://consensys.net/blog/news/programmable-blockchains-in-context-ethereum-smart-contracts-and-the-world-computer/ https://medium.com/mercuryo/decentralized-autonomous-organization-as-a-way-to-manage-defi-projects-4905d4789e27 https://medium.com/mercuryo/decentralized-autonomous-organization-as-a-way-to-manage-defi-projects-4905d4789e27 https://medium.com/mercuryo/decentralized-autonomous-organization-as-a-way-to-manage-defi-projects-4905d4789e27 https://www.forbes.com/sites/jeffkauflin/2019/04/26/why-everyone-in-crypto-is-talking-about-defi/?sh=af68fe1723f3 https://www.forbes.com/sites/jeffkauflin/2019/04/26/why-everyone-in-crypto-is-talking-about-defi/?sh=af68fe1723f3 https://www.forbes.com/sites/jeffkauflin/2019/04/26/why-everyone-in-crypto-is-talking-about-defi/?sh=af68fe1723f3 https://hackernoon.com/left-right-political-ideologies-of-hodlers-part-1-faa71ed689e4 https://hackernoon.com/left-right-political-ideologies-of-hodlers-part-1-faa71ed689e4 https://hackernoon.com/left-right-political-ideologies-of-hodlers-part-1-faa71ed689e4 https://cryptoslate.com/how-many-ethereum-in-circulation-asks-the-bitcoin-community/ https://cryptoslate.com/how-many-ethereum-in-circulation-asks-the-bitcoin-community/ https://cryptoslate.com/how-many-ethereum-in-circulation-asks-the-bitcoin-community/ https://www.theatlantic.com/technology/archive/2017/05/cryptocurrency-ponzi-schemes/528624/ https://www.theatlantic.com/technology/archive/2017/05/cryptocurrency-ponzi-schemes/528624/ https://www.theatlantic.com/technology/archive/2017/05/cryptocurrency-ponzi-schemes/528624/ https://cointelegraph.com/news/vitalik-buterin-sets-milestones-on-ethereums-route-to-be-the-world-computer https://cointelegraph.com/news/vitalik-buterin-sets-milestones-on-ethereums-route-to-be-the-world-computer https://cointelegraph.com/news/vitalik-buterin-sets-milestones-on-ethereums-route-to-be-the-world-computer https://www.wired.com/2016/06/the-uncanny-mind-that-built-ethereum https://www.blockchains-expert.com/en/ethereum-the-world-computer https://coindesk.com/no-crypto-isnt-just-for-libertarians-anymore https://www.e-ir.info/2014/06/01/visions-of-a-techno-leviathan-the-politics-of-the-bitcoin-blockchain https://www.e-ir.info/2014/06/01/visions-of-a-techno-leviathan-the-politics-of-the-bitcoin-blockchain http://www.cjc-online.ca sharma, rakesh. (2020, september 12). why is ethereum co-founder proposing a hard cap? investopedia. url: https://www.investopedia.com/news/why-ethereum-cofounder -proposing-hard-cap/#:~:text=ethereum currently doesn’t have, regulated on a yearly basis [december 5, 2020]. swan, melanie. (2015). blockchain: blueprint for a new economy. sebastopol, ca: o’reilly media.  swartz, lana. (2018). what was bitcoin, what will it be? the techno-economic imaginaries of a new money technology. cultural studies, 32(4), 623–650. vidan, gili, & lehdonvirta, vili. (2019). mine the gap: bitcoin and the maintenance of trustlessness. new media & society, 21(1), 42–59. waters, richard. (2016, may 17). automated company raises equivalent of $120m in digital currency. cnbc. url: https://www.cnbc.com/2016/05/17/automated-company-raises-equivalent-of120-million-in-digital-currency.html [may 10, 2019]. wood, gavin. (2018). why we need web 3.0. medium. url: https://medium.com/@gavofyork/whywe-need-web-3-0-5da4f2bf95ab [may 10, 2019]. wright, aaron, & de filippi, primavera. (2015). decentralized blockchain technology and the rise of lex cryptographia [ssrn scholarly paper, id 2580664]. doi: http://dx.doi.org/10.2139/ssrn .2580664 wu, daniel. (2020, january 22). ethereum 2.0: the road to being the world’s computer. medium.com. url: https://medium.com/ppio/ethereum-2-0-the-road-to-being-the-worlds-computer -663d4e6a90f1 [october 5, 2020]. zetzsche, dirk, buckley, ross, arner, douglas, & föhr, linus. (2017). the ico gold rush: it’s a scam, it’s a bubble, it’s a super challenge for regulators. (working paper no. 11). germany, university of luxembourg law. brody & couture the case of ethereum 561 h tt ps :/ /c jc .u tp jo ur na ls .p re ss /d oi /p df /1 0. 22 23 0/ cj c. 20 21 v4 6n 3a 37 01 f ri da y, s ep te m be r 13 , 2 02 4 2: 27 :2 0 a m i p a dd re ss :1 93 .1 .6 4. 12 https://www.investopedia.com/news/why-ethereum-cofounder-proposing-hard-cap/#:~:text=ethereum https://www.investopedia.com/news/why-ethereum-cofounder-proposing-hard-cap/#:~:text=ethereum https://www.investopedia.com/news/why-ethereum-cofounder-proposing-hard-cap/#:~:text=ethereum https://www.cnbc.com/2016/05/17/automated-company-raises-equivalent-of-120-million-in-digital-currency.html https://www.cnbc.com/2016/05/17/automated-company-raises-equivalent-of-120-million-in-digital-currency.html https://medium.com/@gavofyork/why-we-need-web-3-0-5da4f2bf95ab https://medium.com/@gavofyork/why-we-need-web-3-0-5da4f2bf95ab http://dx.doi.org/10.2139/ssrn.2580664 http://dx.doi.org/10.2139/ssrn.2580664 http://dx.doi.org/10.2139/ssrn.2580664 https://medium.com/ppio/ethereum-2-0-the-road-to-being-the-worlds-computer-663d4e6a90f1 https://medium.com/ppio/ethereum-2-0-the-road-to-being-the-worlds-computer-663d4e6a90f1 https://medium.com/ppio/ethereum-2-0-the-road-to-being-the-worlds-computer-663d4e6a90f1 dark mode toggle sharding faq 2017 dec 31 see all posts currently, in all blockchain protocols each node stores the entire state (account balances, contract code and storage, etc.) and processes all transactions. this provides a large amount of security, but greatly limits scalability: a blockchain cannot process more transactions than a single node can. in large part because of this, bitcoin is limited to ~3–7 transactions per second, ethereum to 7–15, etc. however, this poses a question: are there ways to create a new mechanism, where only a small subset of nodes verifies each transaction? as long as there are sufficiently many nodes verifying each transaction that the system is still highly secure, but a sufficiently small percentage of the total validator set so that the system can process many transactions in parallel, could we not split up transaction processing between smaller groups of nodes to greatly increase a blockchain's total throughput? contents what are some trivial but flawed ways of solving the problem? this sounds like there's some kind of scalability trilemma at play. what is this trilemma and can we break through it? what are some moderately simple but only partial ways of solving the scalability problem? what about approaches that do not try to "shard" anything? how does plasma, state channels and other layer 2 technologies fit into the trilemma? state size, history, cryptoeconomics, oh my! define some of these terms before we move further! what is the basic idea behind sharding? what might a basic design of a sharded blockchain look like? what are the challenges here? but doesn't the cap theorem mean that fully secure distributed systems are impossible, and so sharding is futile? what are the security models that we are operating under? how can we solve the single-shard takeover attack in an uncoordinated majority model? how do you actually do this sampling in proof of work, and in proof of stake? how is the randomness for random sampling generated? what are the tradeoffs in making sampling more or less frequent? can we force more of the state to be held user-side so that transactions can be validated without requiring validators to hold all state data? can we split data and execution so that we get the security from rapid shuffling data validation without the overhead of shuffling the nodes that perform state execution? can snarks and starks help? how can we facilitate cross-shard communication? what is the train-and-hotel problem? what are the concerns about sharding through random sampling in a bribing attacker or coordinated choice model? how can we improve on this? what is the data availability problem, and how can we use erasure codes to solve it? can we remove the need to solve data availability with some kind of fancy cryptographic accumulator scheme? so this means that we can actually create scalable sharded blockchains where the cost of making anything bad happen is proportional to the size of the entire validator set? let's walk back a bit. do we actually need any of this complexity if we have instant shuffling? doesn't instant shuffling basically mean that each shard directly pulls validators from the global validator pool so it operates just like a blockchain, and so sharding doesn't actually introduce any new complexities? you mentioned transparent sharding. i'm 12 years old and what is this? what are some advantages and disadvantages of this? how would synchronous cross-shard messages work? what about semi-asynchronous messages? what are guaranteed cross-shard calls? wait, but what if an attacker sends a cross-shard call from every shard into shard x at the same time? wouldn't it be mathematically impossible to include all of these calls in time? congealed gas? this sounds interesting for not just cross-shard operations, but also reliable intra-shard scheduling does guaranteed scheduling, both intra-shard and cross-shard, help against majority collusions trying to censor transactions? could sharded blockchains do a better job of dealing with network partitions? what are the unique challenges of pushing scaling past n = o(c^2)? what about heterogeneous sharding? footnotes what are some trivial but flawed ways of solving the problem? there are three main categories of "easy solutions". the first is to give up on scaling individual blockchains, and instead assume that applications will be split among many different chains. this greatly increases throughput, but at a cost of security: an n-factor increase in throughput using this method necessarily comes with an n-factor decrease in security, as a level of resources 1/n the size of the whole ecosystem will be sufficient to attack any individual chain. hence, it is arguably non-viable for more than small values of n. the second is to simply increase the block size limit. this can work and in some situations may well be the correct prescription, as block sizes may well be constrained more by politics than by realistic technical considerations. but regardless of one's beliefs about any individual case such an approach inevitably has its limits: if one goes too far, then nodes running on consumer hardware will drop out, the network will start to rely exclusively on a very small number of supercomputers running the blockchain, which can lead to great centralization risk. the third is "merge mining", a technique where there are many chains, but all chains share the same mining power (or, in proof of stake systems, stake). currently, namecoin gets a large portion of its security from the bitcoin blockchain by doing this. if all miners participate, this theoretically can increase throughput by a factor of n without compromising security. however, this also has the problem that it increases the computational and storage load on each miner by a factor of n, and so in fact this solution is simply a stealthy form of block size increase. this sounds like there's some kind of scalability trilemma at play. what is this trilemma and can we break through it? the trilemma claims that blockchain systems can only at most have two of the following three properties: decentralization (defined as the system being able to run in a scenario where each participant only has access to o(c) resources, i.e. a regular laptop or small vps) scalability (defined as being able to process o(n) > o(c) transactions) security (defined as being secure against attackers with up to o(n) resources) in the rest of this document, we'll continue using c to refer to the size of computational resources (including computation, bandwidth and storage) available to each node, and n to refer to the size of the ecosystem in some abstract sense; we assume that transaction load, state size, and the market cap of a cryptocurrency are all proportional to n. the key challenge of scalability is finding a way to achieve all three at the base layer. what are some moderately simple but only partial ways of solving the scalability problem? many sharding proposals (e.g. this early bft sharding proposal from loi luu et al at nus, more recent application of similar ideas in zilliqa, as well as this merklix tree1 approach that has been suggested for bitcoin) attempt to either only shard transaction processing or only shard state, without touching the other2. these efforts can lead to some gains in efficiency, but they run into the fundamental problem that they only solve one of the two bottlenecks. we want to be able to process 10,000+ transactions per second without either forcing every node to be a supercomputer or forcing every node to store a terabyte of state data, and this requires a comprehensive solution where the workloads of state storage, transaction processing and even transaction downloading and re-broadcasting at are all spread out across nodes. particularly, the p2p network needs to also be modified to ensure that not every node receives all information from every other node. what about approaches that do not try to "shard" anything? bitcoin-ng can increase scalability somewhat by means of an alternative blockchain design that makes it much safer for the network if nodes are spending large portions of their cpu time verifying blocks. in simple pow blockchains, there are high centralization risks and the safety of consensus is weakened if capacity is increased to the point where more than about 5% of nodes' cpu time is spent verifying blocks; bitcoin-ng's design alleviates this problem. however, this can only increase the scalability of transaction capacity by a constant factor of perhaps 5-50x3,4, and does not increase the scalability of state. that said, bitcoin-ng-style approaches are not mutually exclusive with sharding, and the two can certainly be implemented at the same time. channel-based strategies (lightning network, raiden, etc) can scale transaction capacity by a constant factor but cannot scale state storage, and also come with their own unique sets of tradeoffs and limitations particularly involving denial-of-service attacks. on-chain scaling via sharding (plus other techniques) and off-chain scaling via channels are arguably both necessary and complementary. there exist approaches that use advanced cryptography, such as mimblewimble and strategies based on zk-snarks (eg. coda), to solve one specific part of the scaling problem: initial full node synchronization. instead of verifying the entire history from genesis, nodes could verify a cryptographic proof that the current state legitimately follows from the history. these approaches do solve a legitimate problem, but they are not a substitute for sharding, as they do not remove the need for nodes to download and verify very large amounts of data to stay on the chain in real time. how does plasma, state channels and other layer 2 technologies fit into the trilemma? in the event of a large attack on plasma subchains, all users of the plasma subchains would need to withdraw back to the root chain. if plasma has o(n) users, then this will require o(n) transactions, and so o(n / c) time to process all of the withdrawals. if withdrawal delays are fixed to some d (i.e. the naive implementation), then as soon as n > c * d, there will not be enough space in the blockchain to process all withdrawals in time, and so the system will be insecure; in this mode, plasma should be viewed as increasing scalability only by a (possibly large) constant factor. if withdrawal delays are flexible, so they automatically extend if there are many withdrawals being made, then this means that as n increases further and further, the amount of time that an attacker can force everyone's funds to get locked up increases, and so the level of "security" of the system decreases further and further in a certain sense, as extended denial of access can be viewed as a security failure, albeit one milder than total loss of access. however, this is a different direction of tradeoff from other solutions, and arguably a much milder tradeoff, hence why plasma subchains are nevertheless a large improvement on the status quo. note that there is one design that states that: "given a malicious operator (the worst case), the system degrades to an on-chain token. a malicious operator cannot steal funds and cannot deprive people of their funds for any meaningful amount of time."—https://ethresear.ch/t/roll-up-roll-back-snark-side-chain-17000-tps/3675. see also here for related information. state channels have similar properties, though with different tradeoffs between versatility and speed of finality. other layer 2 technologies include truebit off-chain interactive verification of execution and raiden, which is another organisation working on state channels. proof of stake with casper (which is layer 1) would also improve scaling—it is more decentralizable, not requiring a computer that is able to mine, which tends towards centralized mining farms and institutionalized mining pools as difficulty increases and the size of the state of the blockchain increases. sharding is different to state channels and plasma in that periodically notaries are pseudo-randomly assigned to vote on the validity of collations (analogous to blocks, but without an evm state transition function in phase 1), then these collations are accepted into the main chain after the votes are verified by a committee on the main chain, via a sharding manager contract on the main chain. in phase 5 (see the roadmap for details), shards are tightly coupled to the main chain, so that if any shard or the main chain is invalid, the whole network is invalid. there are other differences between each mechanism, but at a high level, plasma, state channels and truebit are off-chain for an indefinite interval, connect to the main chain at the smart contract, layer 2 level, while they can draw back into and open up from the main chain, whereas shards are regularly linked to the main chain via consensus in-protocol. see also these tweets from vlad. state size, history, cryptoeconomics, oh my! define some of these terms before we move further! state: a set of information that represents the "current state" of a system; determining whether or not a transaction is valid, as well as the effect of a transaction, should in the simplest model depend only on state. examples of state data include the utxo set in bitcoin, balances + nonces + code + storage in ethereum, and domain name registry entries in namecoin. history: an ordered list of all transactions that have taken place since genesis. in a simple model, the present state should be a deterministic function of the genesis state and the history. transaction: an object that goes into the history. in practice, a transaction represents an operation that some user wants to make, and is cryptographically signed. in some systems transactions are called blobs, to emphasize the fact that in these systems these objects may contain arbitrary data and may not in all cases represent an attempt to perform some operation in the protocol. state transition function: a function that takes a state, applies a transaction and outputs a new state. the computation involved may involve adding and subtracting balances from accounts specified by the transaction, verifying digital signatures and running contract code. merkle tree: a cryptographic hash tree structure that can store a very large amount of data, where authenticating each individual piece of data only takes o(log(n)) space and time. see here for details. in ethereum, the transaction set of each block, as well as the state, is kept in a merkle tree, where the roots of the trees are committed to in a block. receipt: an object that represents an effect of a transaction that is not directly stored in the state, but which is still stored in a merkle tree and committed to in a block header or in a special location in the state so that its existence can later be efficiently proven even to a node that does not have all of the data. logs in ethereum are receipts; in sharded models, receipts are used to facilitate asynchronous cross-shard communication. light client: a way of interacting with a blockchain that only requires a very small amount (we'll say o(1), though o(log(c)) may also be accurate in some cases) of computational resources, keeping track of only the block headers of the chain by default and acquiring any needed information about transactions, state or receipts by asking for and verifying merkle proofs of the relevant data on an as-needed basis. state root: the root hash of the merkle tree representing the state5 the ethereum 1.0 state tree, and how the state root fits into the block structure what is the basic idea behind sharding? we split the state and history up into k = o(n / c) partitions that we call "shards". for example, a sharding scheme on ethereum might put all addresses starting with 0x00 into one shard, all addresses starting with 0x01 into another shard, etc. in the simplest form of sharding, each shard also has its own transaction history, and the effect of transactions in some shard k are limited to the state of shard k. one simple example would be a multi-asset blockchain, where there are k shards and each shard stores the balances and processes the transactions associated with one particular asset. in more advanced forms of sharding, some form of cross-shard communication capability, where transactions on one shard can trigger events on other shards, is also included. what might a basic design of a sharded blockchain look like? a simple approach is as follows. for simplicity, this design keeps track of data blobs only; it does not attempt to process a state transition function. there exists a set of validators (ie. proof of stake nodes), who randomly get assigned the right to create shard blocks. during each slot (eg. an 8-second period of time), for each k in [0...999] a random validator gets selected, and given the right to create a block on "shard k", which might contain up to, say, 32 kb of data. also, for each k, a set of 100 validators get selected as attesters. the header of a block together with at least 67 of the attesting signatures can be published as an object that gets included in the "main chain" (also called a beacon chain). note that there are now several "levels" of nodes that can exist in such a system: super-full node downloads the full data of the beacon chain and every shard block referenced in the beacon chain. top-level node processes the beacon chain blocks only, including the headers and signatures of the shard blocks, but does not download all the data of the shard blocks. single-shard node acts as a top-level node, but also fully downloads and verifies every collation on some specific shard that it cares more about. light node downloads and verifies the block headers of main chain blocks only; does not process any collation headers or transactions unless it needs to read some specific entry in the state of some specific shard, in which case it downloads the merkle branch to the most recent collation header for that shard and from there downloads the merkle proof of the desired value in the state. what are the challenges here? single-shard takeover attacks what if an attacker takes over the majority of the validators responsible for attesting to one particular block, either to (respectively) prevent any collations from getting enough signatures or, worse, to submit collations that are invalid? state transition execution single-shard takeover attacks are typically prevented with random sampling schemes, but such schemes also make it more difficult for validators to compute state roots, as they cannot have up-to-date state information for every shard that they could be assigned to. how do we ensure that light clients can still get accurate information about the state? fraud detection if an invalid collation or state claim does get made, how can nodes (including light nodes) be reliably informed of this so that they can detect the fraud and reject the collation if it is truly fraudulent? cross shard communication the above design supports no cross-shard communication. how do we add cross-shard communication safely? the data availability problem as a subset of fraud detection, what about the specific case where data is missing from a collation? superquadratic sharding in the special case where n > c^2, in the simple design given above there would be more than o(c) collation headers, and so an ordinary node would not be able to process even just the top-level blocks. hence, more than two levels of indirection between transactions and top-level block headers are required (i.e. we need "shards of shards"). what is the simplest and best way to do this? however, the effect of a transaction may depend on events that earlier took place in other shards; a canonical example is transfer of money, where money can be moved from shard i to shard j by first creating a "debit" transaction that destroys coins in shard i, and then creating a "credit" transaction that creates coins in shard j, pointing to a receipt created by the debit transaction as proof that the credit is legitimate. but doesn't the cap theorem mean that fully secure distributed systems are impossible, and so sharding is futile? the cap theorem is a result that has to do with distributed consensus; a simple statement is: "in the cases that a network partition takes place, you have to choose either consistency or availability, you cannot have both". the intuitive argument is simple: if the network splits in half, and in one half i send a transaction "send my 10 coins to a" and in the other i send a transaction "send my 10 coins to b", then either the system is unavailable, as one or both transactions will not be processed, or it becomes inconsistent, as one half of the network will see the first transaction completed and the other half will see the second transaction completed. note that the cap theorem has nothing to do with scalability; it applies to any situation where multiple nodes need to agree on a value, regardless of the amount of data that they are agreeing on. all existing decentralized systems have found some compromise between availability and consistency; sharding does not make anything fundamentally harder in this respect. what are the security models that we are operating under? there are several competing models under which the safety of blockchain designs is evaluated: honest majority (or honest supermajority): we assume that there is some set of validators and up to 50% (or 33% or 25%) of those validators are controlled by an attacker, and the remaining validators honestly follow the protocol. honest majority models can have non-adaptive or adaptive adversaries; an adversary is adaptive if they can quickly choose which portion of the validator set to "corrupt", and non-adaptive if they can only make that choice far ahead of time. note that the honest majority assumption may be higher for notary committees with a 61% honesty assumption. uncoordinated majority: we assume that all validators are rational in a game-theoretic sense (except the attacker, who is motivated to make the network fail in some way), but no more than some fraction (often between 25% and 50%) are capable of coordinating their actions. coordinated choice: we assume that most or all validators are controlled by the same actor, or are fully capable of coordinating on the economically optimal choice between themselves. we can talk about the cost to the coalition (or profit to the coalition) of achieving some undesirable outcome. bribing attacker model: we take the uncoordinated majority model, but instead of making the attacker be one of the participants, the attacker sits outside the protocol, and has the ability to bribe any participants to change their behavior. attackers are modeled as having a budget, which is the maximum that they are willing to pay, and we can talk about their cost, the amount that they end up paying to disrupt the protocol equilibrium. bitcoin proof of work with eyal and sirer's selfish mining fix is robust up to 50% under the honest majority assumption, and up to ~23.21% under the uncoordinated majority assumption. schellingcoin is robust up to 50% under the honest majority and uncoordinated majority assumptions, has ε (i.e. slightly more than zero) cost of attack in a coordinated choice model, and has a p + ε budget requirement and ε cost in a bribing attacker model due to p + epsilon attacks. hybrid models also exist; for example, even in the coordinated choice and bribing attacker models, it is common to make an honest minority assumption that some portion (perhaps 1-15%) of validators will act altruistically regardless of incentives. we can also talk about coalitions consisting of between 50-99% of validators either trying to disrupt the protocol or harm other validators; for example, in proof of work, a 51%-sized coalition can double its revenue by refusing to include blocks from all other miners. the honest majority model is arguably highly unrealistic and has already been empirically disproven see bitcoin's spv mining fork for a practical example. it proves too much: for example, an honest majority model would imply that honest miners are willing to voluntarily burn their own money if doing so punishes attackers in some way. the uncoordinated majority assumption may be realistic; there is also an intermediate model where the majority of nodes is honest but has a budget, so they shut down if they start to lose too much money. the bribing attacker model has in some cases been criticized as being unrealistically adversarial, although its proponents argue that if a protocol is designed with the bribing attacker model in mind then it should be able to massively reduce the cost of consensus, as 51% attacks become an event that could be recovered from. we will evaluate sharding in the context of both uncoordinated majority and bribing attacker models. bribing attacker models are similar to maximally-adaptive adversary models, except that the adversary has the additional power that it can solicit private information from all nodes; this distinction can be crucial, for example algorand is secure under adaptive adversary models but not bribing attacker models because of how it relies on private information for random selection. how can we solve the single-shard takeover attack in an uncoordinated majority model? in short, random sampling. each shard is assigned a certain number of notaries (e.g. 150), and the notaries that approve collations on each shard are taken from the sample for that shard. samples can be reshuffled either semi-frequently (e.g. once every 12 hours) or maximally frequently (i.e. there is no real independent sampling process, notaries are randomly selected for each shard from a global pool every block). sampling can be explicit, as in protocols that choose specifically sized "committees" and ask them to vote on the validity and availability of specific collations, or it can be implicit, as in the case of "longest chain" protocols where nodes pseudorandomly assigned to build on specific collations and are expected to "windback verify" at least n ancestors of the collation they are building on. the result is that even though only a few nodes are verifying and creating blocks on each shard at any given time, the level of security is in fact not much lower, in an honest or uncoordinated majority model, than what it would be if every single node was verifying and creating blocks. the reason is simple statistics: if you assume a ~67% honest supermajority on the global set, and if the size of the sample is 150, then with 99.999% probability the honest majority condition will be satisfied on the sample. if you assume a 75% honest supermajority on the global set, then that probability increases to 99.999999998% (see here for calculation details). hence, at least in the honest / uncoordinated majority setting, we have: decentralization (each node stores only o(c) data, as it's a light client in o(c) shards and so stores o(1) * o(c) = o(c) data worth of block headers, as well as o(c) data corresponding to the recent history of one or several shards that it is assigned to at the present time) scalability (with o(c) shards, each shard having o(c) capacity, the maximum capacity is n = o(c^2)) security (attackers need to control at least ~33% of the entire o(n)-sized validator pool in order to stand a chance of taking over the network). in the bribing attacker model (or in the "very very adaptive adversary" model), things are not so easy, but we will get to this later. note that because of the imperfections of sampling, the security threshold does decrease from 50% to ~30-40%, but this is still a surprisingly low loss of security for what may be a 100-1000x gain in scalability with no loss of decentralization. how do you actually do this sampling in proof of work, and in proof of stake? in proof of stake, it is easy. there already is an "active validator set" that is kept track of in the state, and one can simply sample from this set directly. either an in-protocol algorithm runs and chooses 150 validators for each shard, or each validator independently runs an algorithm that uses a common source of randomness to (provably) determine which shard they are at any given time. note that it is very important that the sampling assignment is "compulsory"; validators do not have a choice of what shard they go into. if validators could choose, then attackers with small total stake could concentrate their stake onto one shard and attack it, thereby eliminating the system's security. in proof of work, it is more difficult, as with "direct" proof of work schemes one cannot prevent miners from applying their work to a given shard. it may be possible to use proof-of-file-access forms of proof of work to lock individual miners to individual shards, but it is hard to ensure that miners cannot quickly download or generate data that can be used for other shards and thus circumvent such a mechanism. the best known approach is through a technique invented by dominic williams called "puzzle towers", where miners first perform proof of work on a common chain, which then inducts them into a proof of stake-style validator pool, and the validator pool is then sampled just as in the proof-of-stake case. one possible intermediate route might look as follows. miners can spend a large (o(c)-sized) amount of work to create a new "cryptographic identity". the precise value of the proof of work solution then chooses which shard they have to make their next block on. they can then spend an o(1)-sized amount of work to create a block on that shard, and the value of that proof of work solution determines which shard they can work on next, and so on8. note that all of these approaches make proof of work "stateful" in some way; the necessity of this is fundamental. how is the randomness for random sampling generated? first of all, it is important to note that even if random number generation is heavily exploitable, this is not a fatal flaw for the protocol; rather, it simply means that there is a medium to high centralization incentive. the reason is that because the randomness is picking fairly large samples, it is difficult to bias the randomness by more than a certain amount. the simplest way to show this is through the binomial distribution, as described above; if one wishes to avoid a sample of size n being more than 50% corrupted by an attacker, and an attacker has p% of the global stake pool, the chance of the attacker being able to get such a majority during one round is: here's a table for what this probability would look like in practice for various values of n and p: n = 50 n = 100 n = 150 n = 250 p = 0.4 0.0978 0.0271 0.0082 0.0009 p = 0.33 0.0108 0.0004 1.83 * 10-5 3.98 * 10-8 p = 0.25 0.0001 6.63 * 10-8 4.11 * 10-11 1.81 * 10-17 < p = 0.2 2.09 * 10-6 2.14 * 10-11 2.50 * 10-16 3.96 * 10-26 hence, for n >= 150, the chance that any given random seed will lead to a sample favoring the attacker is very small indeed11,12. what this means from the perspective of security of randomness is that the attacker needs to have a very large amount of freedom in choosing the random values order to break the sampling process outright. most vulnerabilities in proof-of-stake randomness do not allow the attacker to simply choose a seed; at worst, they give the attacker many chances to select the most favorable seed out of many pseudorandomly generated options. if one is very worried about this, one can simply set n to a greater value, and add a moderately hard key-derivation function to the process of computing the randomness, so that it takes more than 2100 computational steps to find a way to bias the randomness sufficiently. now, let's look at the risk of attacks being made that try to influence the randomness more marginally, for purposes of profit rather than outright takeover. for example, suppose that there is an algorithm which pseudorandomly selects 1000 validators out of some very large set (each validator getting a reward of $1), an attacker has 10% of the stake so the attacker's average "honest" revenue 100, and at a cost of $1 the attacker can manipulate the randomness to "re-roll the dice" (and the attacker can do this an unlimited number of times). due to the central limit theorem, the standard deviation of the number of samples, and based on other known results in math the expected maximum of n random samples is slightly under m + s * sqrt(2 * log(n)) where m is the mean and s is the standard deviation. hence the reward for manipulating the randomness and effectively re-rolling the dice (i.e. increasing n) drops off sharply, e.g. with 0 re-trials your expected reward is $100, with one re-trial it's $105.5, with two it's $108.5, with three it's $110.3, with four it's $111.6, with five it's $112.6 and with six it's $113.5. hence, after five retrials it stops being worth it. as a result, an economically motivated attacker with ten percent of stake will (socially wastefully) spend $5 to get an additional revenue of $13, for a net surplus of $8. however, this kind of logic assumes that one single round of re-rolling the dice is expensive. many older proof of stake algorithms have a "stake grinding" vulnerability where re-rolling the dice simply means making a computation locally on one's computer; algorithms with this vulnerability are certainly unacceptable in a sharding context. newer algorithms (see the "validator selection" section in the proof of stake faq) have the property that re-rolling the dice can only be done by voluntarily giving up one's spot in the block creation process, which entails giving up rewards and fees. the best way to mitigate the impact of marginal economically motivated attacks on sample selection is to find ways to increase this cost. one method to increase the cost by a factor of sqrt(n) from n rounds of voting is the majority-bit method devised by iddo bentov. another form of random number generation that is not exploitable by minority coalitions is the deterministic threshold signature approach most researched and advocated by dominic williams. the strategy here is to use a deterministic threshold signature to generate the random seed from which samples are selected. deterministic threshold signatures have the property that the value is guaranteed to be the same regardless of which of a given set of participants provides their data to the algorithm, provided that at least ⅔ of participants do participate honestly. this approach is more obviously not economically exploitable and fully resistant to all forms of stake-grinding, but it has several weaknesses: it relies on more complex cryptography (specifically, elliptic curves and pairings). other approaches rely on nothing but the random-oracle assumption for common hash algorithms. it fails when many validators are offline. a desired goal for public blockchains is to be able to survive very large portions of the network simultaneously disappearing, as long as a majority of the remaining nodes is honest; deterministic threshold signature schemes at this point cannot provide this property. it's not secure in a bribing attacker or coordinated majority model where more than 67% of validators are colluding. the other approaches described in the proof of stake faq above still make it expensive to manipulate the randomness, as data from all validators is mixed into the seed and making any manipulation requires either universal collusion or excluding other validators outright. one might argue that the deterministic threshold signature approach works better in consistency-favoring contexts and other approaches work better in availability-favoring contexts. what are the tradeoffs in making sampling more or less frequent? selection frequency affects just how adaptive adversaries can be for the protocol to still be secure against them; for example, if you believe that an adaptive attack (e.g. dishonest validators who discover that they are part of the same sample banding together and colluding) can happen in 6 hours but not less, then you would be okay with a sampling time of 4 hours but not 12 hours. this is an argument in favor of making sampling happen as quickly as possible. the main challenge with sampling taking place every block is that reshuffling carries a very high amount of overhead. specifically, verifying a block on a shard requires knowing the state of that shard, and so every time validators are reshuffled, validators need to download the entire state for the new shard(s) that they are in. this requires both a strong state size control policy (i.e. economically ensuring that the size of the state does not grow too large, whether by deleting old accounts, restricting the rate of creating new accounts or a combination of the two) and a fairly long reshuffling time to work well. currently, the parity client can download and verify a full ethereum state snapshot via "warp-sync" in ~2-8 hours, suggesting that reshuffling periods of a few days but not less are safe; perhaps this could be reduced somewhat by shrinking the state size via storage rent but even still reshuffling periods would need to be long, potentially making the system vulnerable to adaptive adversaries. however, there are ways of completely avoiding the tradeoff, choosing the creator of the next collation in each shard with only a few minutes of warning but without adding impossibly high state downloading overhead. this is done by shifting responsibility for state storage, and possibly even state execution, away from collators entirely, and instead assigning the role to either users or an interactive verification protocol. can we force more of the state to be held user-side so that transactions can be validated without requiring validators to hold all state data? see also: https://ethresear.ch/t/the-stateless-client-concept/172 the techniques here tend to involve requiring users to store state data and provide merkle proofs along with every transaction that they send. a transaction would be sent along with a merkle proof-of-correct-execution (or "witness"), and this proof would allow a node that only has the state root to calculate the new state root. this proof-of-correct-execution would consist of the subset of objects in the trie that would need to be traversed to access and verify the state information that the transaction must verify; because merkle proofs are o(log(n)) sized, the proof for a transaction that accesses a constant number of objects would also be o(log(n)) sized. the subset of objects in a merkle tree that would need to be provided in a merkle proof of a transaction that accesses several state objects implementing this scheme in its pure form has two flaws. first, it introduces o(log(n)) overhead (~10-30x in practice), although one could argue that this o(log(n)) overhead is not as bad as it seems because it ensures that the validator can always simply keep state data in memory and thus it never needs to deal with the overhead of accessing the hard drive9. second, it can easily be applied if the addresses that are accessed by a transaction are static, but is more difficult to apply if the addresses in question are dynamic that is, if the transaction execution has code of the form read(f(read(x))) where the address of some state read depends on the execution result of some other state read. in this case, the address that the transaction sender thinks the transaction will be reading at the time that they send the transaction may well differ from the address that is actually read when the transaction is included in a block, and so the merkle proof may be insufficient10. this can be solved with access lists (think: a list of accounts and subsets of storage tries), which specify statically what data transactions can access, so when a miner receives a transaction with a witness they can determine that the witness contains all of the data the transaction could possibly access or modify. however, this harms censorship resistance, making attacks similar in form to the attempted dao soft fork possible. can we split data and execution so that we get the security from rapid shuffling data validation without the overhead of shuffling the nodes that perform state execution? yes. we can create a protocol where we split up validators into three roles with different incentives (so that the incentives do not overlap): proposers or collators, a.k.a. prolators, notaries and executors. prollators are responsible for simply building a chain of collations; while notaries verify that the data in the collations is available. prolators do not need to verify anything state-dependent (e.g. whether or not someone trying to send eth has enough money). executors take the chain of collations agreed to by the prolators as given, and then execute the transactions in the collations sequentially and compute the state. if any transaction included in a collation is invalid, executors simply skip over it. this way, validators that verify availability could be reshuffled instantly, and executors could stay on one shard. there would be a light client protocol that allows light clients to determine what the state is based on claims signed by executors, but this protocol is not a simple majority-voting consensus. rather, the protocol is an interactive game with some similarities to truebit, where if there is great disagreement then light client simply execute specific collations or portions of collations themselves. hence, light clients can get a correct view of the state even if 90% of the executors in the shard are corrupted, making it much safer to allow executors to be very infrequently reshuffled or even permanently shard-specific. choosing what goes in to a collation does require knowing the state of that collation, as that is the most practical way to know what will actually pay transaction fees, but this can be solved by further separating the role of collators (who agree on the history) and proposers (who propose individual collations) and creating a market between the two classes of actors; see here for more discussion on this. however, this approach has since been found to be flawed as per this analysis. can snarks and starks help? yes! one can create a second-level protocol where a snark, stark or similar succinct zero knowledge proof scheme is used to prove the state root of a shard chain, and proof creators can be rewarded for this. that said, shard chains to actually agree on what data gets included into the shard chains in the first place is still required. how can we facilitate cross-shard communication? the easiest scenario to satisfy is one where there are very many applications that individually do not have too many users, and which only very occasionally and loosely interact with each other; in this case, applications can live on separate shards and use cross-shard communication via receipts to talk to each other. this typically involves breaking up each transaction into a "debit" and a "credit". for example, suppose that we have a transaction where account a on shard m wishes to send 100 coins to account b on shard n. the steps would looks as follows: send a transaction on shard m which (i) deducts the balance of a by 100 coins, and (ii) creates a receipt. a receipt is an object which is not saved in the state directly, but where the fact that the receipt was generated can be verified via a merkle proof. wait for the first transaction to be included (sometimes waiting for finalization is required; this depends on the system). send a transaction on shard n which includes the merkle proof of the receipt from (1). this transaction also checks in the state of shard n to make sure that this receipt is "unspent"; if it is, then it increases the balance of b by 100 coins, and saves in the state that the receipt is spent. optionally, the transaction in (3) also saves a receipt, which can then be used to perform further actions on shard m that are contingent on the original operation succeeding. in more complex forms of sharding, transactions may in some cases have effects that spread out across several shards and may also synchronously ask for data from the state of multiple shards. what is the train-and-hotel problem? the following example is courtesy of andrew miller. suppose that a user wants to purchase a train ticket and reserve a hotel, and wants to make sure that the operation is atomic either both reservations succeed or neither do. if the train ticket and hotel booking applications are on the same shard, this is easy: create a transaction that attempts to make both reservations, and throws an exception and reverts everything unless both reservations succeed. if the two are on different shards, however, this is not so easy; even without cryptoeconomic / decentralization concerns, this is essentially the problem of atomic database transactions. with asynchronous messages only, the simplest solution is to first reserve the train, then reserve the hotel, then once both reservations succeed confirm both; the reservation mechanism would prevent anyone else from reserving (or at least would ensure that enough spots are open to allow all reservations to be confirmed) for some period of time. however, this means that the mechanism relies on an extra security assumptions: that cross-shard messages from one shard can get included in another shard within some fixed period of time. with cross-shard synchronous transactions, the problem is easier, but the challenge of creating a sharding solution capable of making cross-shard atomic synchronous transactions is itself decidedly nontrivial; see vlad zamfir's presentation which talks about merge blocks. another solution involves making contracts themselves movable across shards; see the proposed cross-shard locking scheme as well as this proposal where contracts can be "yanked" from one shard to another, allowing two contracts that normally reside on different shards to be temporarily moved to the same shard at which point a synchronous operation between them can happen. what are the concerns about sharding through random sampling in a bribing attacker or coordinated choice model? in a bribing attacker or coordinated choice model, the fact that validators are randomly sampled doesn't matter: whatever the sample is, either the attacker can bribe the great majority of the sample to do as the attacker pleases, or the attacker controls a majority of the sample directly and can direct the sample to perform arbitrary actions at low cost (o(c) cost, to be precise). at that point, the attacker has the ability to conduct 51% attacks against that sample. the threat is further magnified because there is a risk of cross-shard contagion: if the attacker corrupts the state of a shard, the attacker can then start to send unlimited quantities of funds out to other shards and perform other cross-shard mischief. all in all, security in the bribing attacker or coordinated choice model is not much better than that of simply creating o(c) altcoins. how can we improve on this? in the context of state execution, we can use interactive verification protocols that are not randomly sampled majority votes, and that can give correct answers even if 90% of the participants are faulty; see truebit for an example of how this can be done. for data availability, the problem is harder, though there are several strategies that can be used alongside majority votes to solve it. what is the data availability problem, and how can we use erasure codes to solve it? see https://github.com/ethereum/research/wiki/a-note-on-data-availability-and-erasure-coding can we remove the need to solve data availability with some kind of fancy cryptographic accumulator scheme? no. suppose there is a scheme where there exists an object s representing the state (s could possibly be a hash) possibly as well as auxiliary information ("witnesses") held by individual users that can prove the presence of existing state objects (e.g. s is a merkle root, the witnesses are the branches, though other constructions like rsa accumulators do exist). there exists an updating protocol where some data is broadcasted, and this data changes s to change the contents of the state, and also possibly changes witnesses. suppose some user has the witnesses for a set of n objects in the state, and m of the objects are updated. after receiving the update information, the user can check the new status of all n objects, and thereby see which m were updated. hence, the update information itself encoded at least ~m * log(n) bits of information. hence, the update information that everyone needs for receive to implement the effect of m transactions must necessarily be of size o(m). 14 so this means that we can actually create scalable sharded blockchains where the cost of making anything bad happen is proportional to the size of the entire validator set? there is one trivial attack by which an attacker can always burn o(c) capital to temporarily reduce the quality of a shard: spam it by sending transactions with high transaction fees, forcing legitimate users to outbid you to get in. this attack is unavoidable; you could compensate with flexible gas limits, and you could even try "transparent sharding" schemes that try to automatically re-allocate nodes to shards based on usage, but if some particular application is non-parallelizable, amdahl's law guarantees that there is nothing you can do. the attack that is opened up here (reminder: it only works in the zamfir model, not honest/uncoordinated majority) is arguably not substantially worse than the transaction spam attack. hence, we've reached the known limit for the security of a single shard, and there is no value in trying to go further. let's walk back a bit. do we actually need any of this complexity if we have instant shuffling? doesn't instant shuffling basically mean that each shard directly pulls validators from the global validator pool so it operates just like a blockchain, and so sharding doesn't actually introduce any new complexities? kind of. first of all, it's worth noting that proof of work and simple proof of stake, even without sharding, both have very low security in a bribing attacker model; a block is only truly "finalized" in the economic sense after o(n) time (as if only a few blocks have passed, then the economic cost of replacing the chain is simply the cost of starting a double-spend from before the block in question). casper solves this problem by adding its finality mechanism, so that the economic security margin immediately increases to the maximum. in a sharded chain, if we want economic finality then we need to come up with a chain of reasoning for why a validator would be willing to make a very strong claim on a chain based solely on a random sample, when the validator itself is convinced that the bribing attacker and coordinated choice models may be true and so the random sample could potentially be corrupted. you mentioned transparent sharding. i'm 12 years old and what is this? basically, we do not expose the concept of "shards" directly to developers, and do not permanently assign state objects to specific shards. instead, the protocol has an ongoing built-in load-balancing process that shifts objects around between shards. if a shard gets too big or consumes too much gas it can be split in half; if two shards get too small and talk to each other very often they can be combined together; if all shards get too small one shard can be deleted and its contents moved to various other shards, etc. imagine if donald trump realized that people travel between new york and london a lot, but there's an ocean in the way, so he could just take out his scissors, cut out the ocean, glue the us east coast and western europe together and put the atlantic beside the south pole it's kind of like that. what are some advantages and disadvantages of this? developers no longer need to think about shards there's the possibility for shards to adjust manually to changes in gas prices, rather than relying on market mechanics to increase gas prices in some shards more than others there is no longer a notion of reliable co-placement: if two contracts are put into the same shard so that they can interact with each other, shard changes may well end up separating them more protocol complexity the co-placement problem can be mitigated by introducing a notion of "sequential domains", where contracts may specify that they exist in the same sequential domain, in which case synchronous communication between them will always be possible. in this model a shard can be viewed as a set of sequential domains that are validated together, and where sequential domains can be rebalanced between shards if the protocol determines that it is efficient to do so. how would synchronous cross-shard messages work? the process becomes much easier if you view the transaction history as being already settled, and are simply trying to calculate the state transition function. there are several approaches; one fairly simple approach can be described as follows: a transaction may specify a set of shards that it can operate in in order for the transaction to be effective, it must be included at the same block height in all of these shards. transactions within a block must be put in order of their hash (this ensures a canonical order of execution) a client on shard x, if it sees a transaction with shards (x, y), requests a merkle proof from shard y verifying (i) the presence of that transaction on shard y, and (ii) what the pre-state on shard y is for those bits of data that the transaction will need to access. it then executes the transaction and commits to the execution result. note that this process may be highly inefficient if there are many transactions with many different "block pairings" in each block; for this reason, it may be optimal to simply require blocks to specify sister shards, and then calculation can be done more efficiently at a per-block level. this is the basis for how such a scheme could work; one could imagine more complex designs. however, when making a new design, it's always important to make sure that low-cost denial of service attacks cannot arbitrarily slow state calculation down. what about semi-asynchronous messages? vlad zamfir created a scheme by which asynchronous messages could still solve the "book a train and hotel" problem. this works as follows. the state keeps track of all operations that have been recently made, as well as the graph of which operations were triggered by any given operation (including cross-shard operations). if an operation is reverted, then a receipt is created which can then be used to revert any effect of that operation on other shards; those reverts may then trigger their own reverts and so forth. the argument is that if one biases the system so that revert messages can propagate twice as fast as other kinds of messages, then a complex cross-shard transaction that finishes executing in k rounds can be fully reverted in another k rounds. the overhead that this scheme would introduce has arguably not been sufficiently studied; there may be worst-case scenarios that trigger quadratic execution vulnerabilities. it is clear that if transactions have effects that are more isolated from each other, the overhead of this mechanism is lower; perhaps isolated executions can be incentivized via favorable gas cost rules. all in all, this is one of the more promising research directions for advanced sharding. what are guaranteed cross-shard calls? one of the challenges in sharding is that when a call is made, there is by default no hard protocol-provided guarantee that any asynchronous operations created by that call will be made within any particular timeframe, or even made at all; rather, it is up to some party to send a transaction in the destination shard triggering the receipt. this is okay for many applications, but in some cases it may be problematic for several reasons: there may be no single party that is clearly incentivized to trigger a given receipt. if the sending of a transaction benefits many parties, then there could be tragedy-of-the-commons effects where the parties try to wait longer until someone else sends the transaction (i.e. play "chicken"), or simply decide that sending the transaction is not worth the transaction fees for them individually. gas prices across shards may be volatile, and in some cases performing the first half of an operation compels the user to "follow through" on it, but the user may have to end up following through at a much higher gas price. this may be exacerbated by dos attacks and related forms of griefing. some applications rely on there being an upper bound on the "latency" of cross-shard messages (e.g. the train-and-hotel example). lacking hard guarantees, such applications would have to have inefficiently large safety margins. one could try to come up with a system where asynchronous messages made in some shard automatically trigger effects in their destination shard after some number of blocks. however, this requires every client on each shard to actively inspect all other shards in the process of calculating the state transition function, which is arguably a source of inefficiency. the best known compromise approach is this: when a receipt from shard a at height height_a is included in shard b at height height_b, if the difference in block heights exceeds max_height, then all validators in shard b that created blocks from height_a + max_height + 1 to height_b 1 are penalized, and this penalty increases exponentially. a portion of these penalties is given to the validator that finally includes the block as a reward. this keeps the state transition function simple, while still strongly incentivizing the correct behavior. wait, but what if an attacker sends a cross-shard call from every shard into shard x at the same time? wouldn't it be mathematically impossible to include all of these calls in time? correct; this is a problem. here is a proposed solution. in order to make a cross-shard call from shard a to shard b, the caller must pre-purchase "congealed shard b gas" (this is done via a transaction in shard b, and recorded in shard b). congealed shard b gas has a fast demurrage rate: once ordered, it loses 1/k of its remaining potency every block. a transaction on shard a can then send the congealed shard b gas along with the receipt that it creates, and it can be used on shard b for free. shard b blocks allocate extra gas space specifically for these kinds of transactions. note that because of the demurrage rules, there can be at most gas_limit * k worth of congealed gas for a given shard available at any time, which can certainly be filled within k blocks (in fact, even faster due to demurrage, but we may need this slack space due to malicious validators). in case too many validators maliciously fail to include receipts, we can make the penalties fairer by exempting validators who fill up the "receipt space" of their blocks with as many receipts as possible, starting with the oldest ones. under this pre-purchase mechanism, a user that wants to perform a cross-shard operation would first pre-purchase gas for all shards that the operation would go into, over-purchasing to take into account the demurrage. if the operation would create a receipt that triggers an operation that consumes 100000 gas in shard b, the user would pre-buy 100000 * e (i.e. 271818) shard-b congealed gas. if that operation would in turn spend 100000 gas in shard c (i.e. two levels of indirection), the user would need to pre-buy 100000 * e^2 (i.e. 738906) shard-c congealed gas. notice how once the purchases are confirmed, and the user starts the main operation, the user can be confident that they will be insulated from changes in the gas price market, unless validators voluntarily lose large quantities of money from receipt non-inclusion penalties. congealed gas? this sounds interesting for not just cross-shard operations, but also reliable intra-shard scheduling indeed; you could buy congealed shard a gas inside of shard a, and send a guaranteed cross-shard call from shard a to itself. though note that this scheme would only support scheduling at very short time intervals, and the scheduling would not be exact to the block; it would only be guaranteed to happen within some period of time. does guaranteed scheduling, both intra-shard and cross-shard, help against majority collusions trying to censor transactions? yes. if a user fails to get a transaction in because colluding validators are filtering the transaction and not accepting any blocks that include it, then the user could send a series of messages which trigger a chain of guaranteed scheduled messages, the last of which reconstructs the transaction inside of the evm and executes it. preventing such circumvention techniques is practically impossible without shutting down the guaranteed scheduling feature outright and greatly restricting the entire protocol, and so malicious validators would not be able to do it easily. could sharded blockchains do a better job of dealing with network partitions? the schemes described in this document would offer no improvement over non-sharded blockchains; realistically, every shard would end up with some nodes on both sides of the partition. there have been calls (e.g. from ipfs's juan benet) for building scalable networks with the specific goal that networks can split up into shards as needed and thus continue operating as much as possible under network partition conditions, but there are nontrivial cryptoeconomic challenges in making this work well. one major challenge is that if we want to have location-based sharding so that geographic network partitions minimally hinder intra-shard cohesion (with the side effect of having very low intra-shard latencies and hence very fast intra-shard block times), then we need to have a way for validators to choose which shards they are participating in. this is dangerous, because it allows for much larger classes of attacks in the honest/uncoordinated majority model, and hence cheaper attacks with higher griefing factors in the zamfir model. sharding for geographic partition safety and sharding via random sampling for efficiency are two fundamentally different things. second, more thinking would need to go into how applications are organized. a likely model in a sharded blockchain as described above is for each "app" to be on some shard (at least for small-scale apps); however, if we want the apps themselves to be partition-resistant, then it means that all apps would need to be cross-shard to some extent. one possible route to solving this is to create a platform that offers both kinds of shards some shards would be higher-security "global" shards that are randomly sampled, and other shards would be lower-security "local" shards that could have properties such as ultra-fast block times and cheaper transaction fees. very low-security shards could even be used for data-publishing and messaging. what are the unique challenges of pushing scaling past n = o(c^2)? there are several considerations. first, the algorithm would need to be converted from a two-layer algorithm to a stackable n-layer algorithm; this is possible, but is complex. second, n / c (i.e. the ratio between the total computation load of the network and the capacity of one node) is a value that happens to be close to two constants: first, if measured in blocks, a timespan of several hours, which is an acceptable "maximum security confirmation time", and second, the ratio between rewards and deposits (an early computation suggests a 32 eth deposit size and a 0.05 eth block reward for casper). the latter has the consequence that if rewards and penalties on a shard are escalated to be on the scale of validator deposits, the cost of continuing an attack on a shard will be o(n) in size. going above c^2 would likely entail further weakening the kinds of security guarantees that a system can provide, and allowing attackers to attack individual shards in certain ways for extended periods of time at medium cost, although it should still be possible to prevent invalid state from being finalized and to prevent finalized state from being reverted unless attackers are willing to pay an o(n) cost. however, the rewards are large a super-quadratically sharded blockchain could be used as a general-purpose tool for nearly all decentralized applications, and could sustain transaction fees that makes its use virtually free. what about heterogeneous sharding? abstracting the execution engine or allowing multiple execution engines to exist results in being able to have a different execution engine for each shard. due to casper cbc being able to explore the full tradeoff triangle, it is possible to alter the parameters of the consensus engine for each shard to be at any point of the triangle. however, cbc casper has not been implemented yet, and heterogeneous sharding is nothing more than an idea at this stage; the specifics of how it would work has not been designed nor implemented. some shards could be optimized to have fast finality and high throughput, which is important for applications such as eftpos transactions, while maybe most could have a moderate or reasonable amount each of finality, throughput and decentralization (number of validating nodes), and applications that are prone to a high fault rate and thus require high security, such as torrent networks, privacy focused email like proton mail, etc., could optimize for a high decentralization, low finality and high throughput, etc. see also https://twitter.com/vladzamfir/status/932320997021171712 and https://ethresear.ch/t/heterogeneous-sharding/1979/2. footnotes merklix tree == merkle patricia tree later proposals from the nus group do manage to shard state; they do this via the receipt and state-compacting techniques that i describe in later sections in this document. (this is vitalik buterin writing as the creator of this wiki.) there are reasons to be conservative here. particularly, note that if an attacker comes up with worst-case transactions whose ratio between processing time and block space expenditure (bytes, gas, etc) is much higher than usual, then the system will experience very low performance, and so a safety factor is necessary to account for this possibility. in traditional blockchains, the fact that block processing only takes ~1-5% of block time has the primary role of protecting against centralization risk but serves double duty of protecting against denial of service risk. in the specific case of bitcoin, its current worst-case known quadratic execution vulnerability arguably limits any scaling at present to ~5-10x, and in the case of ethereum, while all known vulnerabilities are being or have been removed after the denial-of-service attacks, there is still a risk of further discrepancies particularly on a smaller scale. in bitcoin ng, the need for the former is removed, but the need for the latter is still there. a further reason to be cautious is that increased state size corresponds to reduced throughput, as nodes will find it harder and harder to keep state data in ram and so need more and more disk accesses, and databases, which often have an o(log(n)) access time, will take longer and longer to access. this was an important lesson from the last ethereum denial-of-service attack, which bloated the state by ~10 gb by creating empty accounts and thereby indirectly slowed processing down by forcing further state accesses to hit disk instead of ram. in sharded blockchains, there may not necessarily be in-lockstep consensus on a single global state, and so the protocol never asks nodes to try to compute a global state root; in fact, in the protocols presented in later sections, each shard has its own state, and for each shard there is a mechanism for committing to the state root for that shard, which represents that shard's state #mega if a non-scalable blockchain upgrades into a scalable blockchain, the author's recommended path is that the old chain's state should simply become a single shard in the new chain. for this to be secure, some further conditions must be satisfied; particularly, the proof of work must be non-outsourceable in order to prevent the attacker from determining which other miners' identities are available for some given shard and mining on top of those. recent ethereum denial-of-service attacks have proven that hard drive access is a primary bottleneck to blockchain scalability. you could ask: well why don't validators fetch merkle proofs just-in-time? answer: because doing so is a ~100-1000ms roundtrip, and executing an entire complex transaction within that time could be prohibitive. one hybrid solution that combines the normal-case efficiency of small samples with the greater robustness of larger samples is a multi-layered sampling scheme: have a consensus between 50 nodes that requires 80% agreement to move forward, and then only if that consensus fails to be reached then fall back to a 250-node sample. n = 50 with an 80% threshold has only a 8.92 * 10-9 failure rate even against attackers with p = 0.4, so this does not harm security at all under an honest or uncoordinated majority model. the probabilities given are for one single shard; however, the random seed affects o(c) shards and the attacker could potentially take over any one of them. if we want to look at o(c) shards simultaneously, then there are two cases. first, if the grinding process is computationally bounded, then this fact does not change the calculus at all, as even though there are now o(c) chances of success per round, checking success takes o(c) times as much work. second, if the grinding process is economically bounded, then this indeed calls for somewhat higher safety factors (increasing n by 10-20 should be sufficient) although it's important to note that the goal of an attacker in a profit-motivated manipulation attack is to increase their participation across all shards in any case, and so that is the case that we are already investigating. see parity's polkadotpaper for further description of how their "fishermen" concept works. for up-to-date info and code for polkadot, see here. thanks to justin drake for pointing me to cryptographic accumulators, as well as this paper that gives the argument for the impossibility of sublinear batching. see also this thread: https://ethresear.ch/t/accumulators-scalability-of-utxo-blockchains-and-data-availability/176 further reading related to sharding, and more generally scalability and research, is available here and here. ethereum: a next-generation smart contract and decentralized application platform. when satoshi nakamoto first set the bitcoin blockchain into motion in january 2009, he was simultaneously introducing two radical and untested concepts. the first is the "bitcoin", a decentralized peer-to-peer online currency that maintains a value without any backing, intrinsic value or central issuer. so far, the "bitcoin" as a currency unit has taken up the bulk of the public attention, both in terms of the political aspects of a currency without a central bank and its extreme upward and downward volatility in price. however, there is also another, equally important, part to satoshi's grand experiment: the concept of a proof of work-based blockchain to allow for public agreement on the order of transactions. bitcoin as an application can be described as a first-to-file system: if one entity has 50 btc, and simultaneously sends the same 50 btc to a and to b, only the transaction that gets confirmed first will process. there is no intrinsic way of determining from two transactions which came earlier, and for decades this stymied the development of decentralized digital currency. satoshi's blockchain was the first credible decentralized solution. and now, attention is rapidly starting to shift toward this second part of bitcoin's technology, and how the blockchain concept can be used for more than just money. commonly cited applications include using on-blockchain digital assets to represent custom currencies and financial instruments ("colored coins"), the ownership of an underlying physical device ("smart property"), non-fungible assets such as domain names ("namecoin") as well as more advanced applications such as decentralized exchange, financial derivatives, peer-to-peer gambling and on-blockchain identity and reputation systems. another important area of inquiry is "smart contracts" systems which automatically move digital assets according to arbitrary pre-specified rules. for example, one might have a treasury contract of the form "a can withdraw up to x currency units per day, b can withdraw up to y per day, a and b together can withdraw anything, and a can shut off b's ability to withdraw". the logical extension of this is decentralized autonomous organizations (daos) long-term smart contracts that contain the assets and encode the bylaws of an entire organization. what ethereum intends to provide is a blockchain with a built-in fully fledged turing-complete programming language that can be used to create "contracts" that can be used to encode arbitrary state transition functions, allowing users to create any of the systems described above, as well as many others that we have not yet imagined, simply by writing up the logic in a few lines of code. page 1 ethereum.org ethereum: a next-generation smart contract and decentralized application platform. by vitalik buterin (2014). http://www.google.com/url?q=http%3a%2f%2fbitcoinmagazine.com%2f8640%2fan-exploration-of-intrinsic-value-what-it-is-why-bitcoin-doesnt-have-it-and-why-bitcoin-does-have-it%2f&sa=d&sntz=1&usg=afqjcnhihj5joz_hutamkn106c0c2k7t7w https://docs.google.com/a/buterin.com/document/d/1ankp_cvztcmlizw4dvsw6m8q2jc0lizrtluowu2z1be/edit http://www.google.com/url?q=http%3a%2f%2fbitcoinmagazine.com%2f7050%2fbootstrapping-a-decentralized-autonomous-corporation-part-i%2f&sa=d&sntz=1&usg=afqjcnf8ximt3yrvaid-x0dofldgk1jsjg table of contents ● history ○ bitcoin as a state transition system ○ mining ○ merkle trees ○ alternative blockchain applications ○ scripting ● ethereum ○ ethereum accounts ○ messages and transactions ○ ethereum state transition function ○ code execution ○ blockchain and mining ● applications ○ token systems ○ financial derivatives ○ identity and reputation systems ○ decentralized file storage ○ decentralized autonomous organizations ○ further applications ● miscellanea and concerns ○ modified ghost implementation ○ fees ○ computation and turing-completeness ○ currency and issuance ○ mining centralization ○ scalability ● putting it all together: decentralized applications ● conclusion ● references and further reading page 2 ethereum.org https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-white-paper%23mining&sa=d&sntz=1&usg=afqjcnhxhez8d-jmiogwh50cfybaeho3rg https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-white-paper%23merkle-trees&sa=d&sntz=1&usg=afqjcnh-5ttar5zzahkrfajdxdrqsm29xq https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-white-paper%23alternative-blockchain-applications&sa=d&sntz=1&usg=afqjcng35ozh-fsz_ga6f0-lmlptajuv-g https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-white-paper%23scripting&sa=d&sntz=1&usg=afqjcnhnivniihenkeorrqs6_gnqrzkpew page 3 ethereum.org history the concept of decentralized digital currency, as well as alternative applications like property registries, has been around for decades. the anonymous e-cash protocols of the 1980s and the 1990s, mostly reliant on a cryptographic primitive known as chaumian blinding, provided a currency with a high degree of privacy, but the protocols largely failed to gain traction because of their reliance on a centralized intermediary. in 1998, wei dai's b-money became the first proposal to introduce the idea of creating money through solving computational puzzles as well as decentralized consensus, but the proposal was scant on details as to how decentralized consensus could actually be implemented. in 2005, hal finney introduced a concept of "reusable proofs of work", a system which uses ideas from b-money together with adam back's computationally difficult hashcash puzzles to create a concept for a cryptocurrency, but once again fell short of the ideal by relying on trusted computing as a backend. because currency is a first-to-file application, where the order of transactions is often of critical importance, decentralized currencies require a solution to decentralized consensus. the main roadblock that all pre-bitcoin currency protocols faced is the fact that, while there had been plenty of research on creating secure byzantine-fault-tolerant multiparty consensus systems for many years, all of the protocols described were solving only half of the problem. the protocols assumed that all participants in the system were known, and produced security margins of the form "if n parties participate, then the system can tolerate up to n/4 malicious actors". the problem is, however, that in an anonymous setting such security margins are vulnerable to sybil attacks, where a single attacker creates thousands of simulated nodes on a server or botnet and uses these nodes to unilaterally secure a majority share. the innovation provided by satoshi is the idea of combining a very simple decentralized consensus protocol, based on nodes combining transactions into a "block" every ten minutes creating an ever-growing blockchain, with proof of work as a mechanism through which nodes gain the right to participate in the system. while nodes with a large amount of computational power do have proportionately greater influence, coming up with more computational power than the entire network combined is much harder than simulating a million nodes. despite the bitcoin blockchain model's crudeness and simplicity, it has proven to be good enough, and would over the next five years become the bedrock of over two hundred currencies and protocols around the world. page 4 ethereum.org http://www.google.com/url?q=http%3a%2f%2fwww.weidai.com%2fbmoney.txt&sa=d&sntz=1&usg=afqjcneythbdnq0vi1g--o4hwyuoscbiqa http://www.google.com/url?q=http%3a%2f%2fwww.finney.org%2f%257ehal%2frpow%2f&sa=d&sntz=1&usg=afqjcnfzg0g7mwxclfki1yqvmfqeeipe2a bitcoin as a state transition system from a technical standpoint, the bitcoin ledger can be thought of as a state transition system, where there is a "state" consisting of the ownership status of all existing bitcoins and a "state transition function" that takes a state and a transaction and outputs a new state which is the result. in a standard banking system, for example, the state is a balance sheet, a transaction is a request to move $x from a to b, and the state transition function reduces the value in a's account by $x and increases the value in b's account by $x. if a's account has less than $x in the first place, the state transition function returns an error. hence, one can formally define: apply(s,tx) ­> s' or error in the banking system defined above: apply({ alice: $50, bob: $50 },"send $20 from alice to bob") = { alice: $30,                              bob: $70 } but: apply({ alice: $50, bob: $50 },"send $70 from alice to bob") = error    the "state" in bitcoin is the collection of all coins (technically, "unspent transaction outputs" or utxo) that have been minted and not yet spent, with each utxo having a denomination and an owner (defined by a 20-byte address which is essentially a cryptographic public key[1]). a transaction contains one or more inputs, with each input containing a reference to an existing utxo and a cryptographic signature produced by the private key associated with the owner's address, and one or more outputs, with each output containing a new utxo to be added to the state. page 5 ethereum.org the state transition function apply(s,tx) ­> s' can be defined roughly as follows: 1. for each input in tx: i. if the referenced utxo is not in s, return an error. ii. if the provided signature does not match the owner of the utxo, return an error. 2. if the sum of the denominations of all input utxo is less than the sum of the denominations of all output utxo, return an error. 3. return s with all input utxo removed and all output utxo added. the first half of the first step prevents transaction senders from spending coins that do not exist, the second half of the first step prevents transaction senders from spending other people's coins, and the second step enforces conservation of value. in order to use this for payment, the protocol is as follows. suppose alice wants to send 11.7 btc to bob. first, alice will look for a set of available utxo that she owns that totals up to at least 11.7 btc. realistically, alice will not be able to get exactly 11.7 btc; say that the smallest she can get is 6+4+2=12. she then creates a transaction with those three inputs and two outputs. the first output will be 11.7 btc with bob's address as its owner, and the second output will be the remaining 0.3 btc "change", with the owner being alice herself. mining if we had access to a trustworthy centralized service, this system would be trivial to implement; it could simply be coded exactly as described. however, with bitcoin we are trying to build a decentralized currency system, so we will need to combine the state transition system with a consensus system in order to ensure that everyone agrees on the order of transactions. bitcoin's decentralized consensus process requires nodes in the network to continuously attempt to produce packages of transactions called "blocks". the network is intended to produce roughly one block every ten minutes, with each block containing a timestamp, a nonce, a reference to (ie. hash of) the page 6 ethereum.org previous block and a list of all of the transactions that have taken place since the previous block. over time, this creates a persistent, ever-growing, "blockchain" that constantly updates to represent the latest state of the bitcoin ledger. the algorithm for checking if a block is valid, expressed in this paradigm, is as follows: 1. check if the previous block referenced by the block exists and is valid 2. check that the timestamp of the block is greater than that of the previous block[2] and less than 2 hours into the future. 3. check that the proof of work on the block is valid. 4. let s[0] be the state at the end of the previous block. 5. suppose tx is the block's transaction list with n transactions. for all i in 0...n-1, sets[i+1] = apply(s[i],tx[i]) if any application returns an error, exit and return false. 6. return true, and register s[n] as the state at the end of this block essentially, each transaction in the block must provide a state transition that is valid. note that the state is not encoded in the block in any way; it is purely an abstraction to be remembered by the validating node and can only be (securely) computed for any block by starting from the genesis state and sequentially applying every transaction in every block. additionally, note that the order in which the miner includes transactions into the block matters; if there are two transactions a and b in a block such that b spends a utxo created by a, then the block will be valid if a comes before b but not otherwise. the interesting part of the block validation algorithm is the concept of "proof of work": the condition is that the sha256 hash of every block, treated as a 256-bit number, must be less than a dynamically adjusted target, which as of the time of this writing is approximately 2190. the purpose of this is to make block creation computationally "hard", thereby preventing sybil attackers from remaking the entire blockchain in their favor. because sha256 is designed to be a completely unpredictable pseudorandom function, the only way to create a valid block is simply trial and error, repeatedly incrementing the nonce and seeing if the new hash matches. at the current target of 2192, this means an average of 264 tries; in general, the target is recalibrated by the network every 2016 blocks so that on average a new block is produced by some node in the network every ten minutes. in order to compensate miners for this computational work, the miner of every block is entitled to include a transaction giving themselves 25 btc out of nowhere. additionally, if any transaction has a higher total denomination in its inputs than in its outputs, the difference also goes to the miner as a "transaction fee". incidentally, this is also the only mechanism by which btc are issued; the genesis state contained no coins at all. page 7 ethereum.org in order to better understand the purpose of mining, let us examine what happens in the event of a malicious attacker. since bitcoin's underlying cryptography is known to be secure, the attacker will target the one part of the bitcoin system that is not protected by cryptography directly: the order of transactions. the attacker's strategy is simple: 1. send 100 btc to a merchant in exchange for some product (preferably a rapid-delivery digital good) 2. wait for the delivery of the product 3. produce another transaction sending the same 100 btc to himself 4. try to convince the network that his transaction to himself was the one that came first. once step (1) has taken place, after a few minutes some miner will include the transaction in a block, say block number 270000. after about one hour, five more blocks will have been added to the chain after that block, with each of those blocks indirectly pointing to the transaction and thus "confirming" it. at this point, the merchant will accept the payment as finalized and deliver the product; since we are assuming this is a digital good, delivery is instant. now, the attacker creates another transaction sending the 100 btc to himself. if the attacker simply releases it into the wild, the transaction will not be processed; miners will attempt to run apply(s,tx) and notice that tx consumes a utxo which is no longer in the state. so instead, the attacker creates a "fork" of the blockchain, starting by mining another version of block 270000 pointing to the same block 269999 as a parent but with the new transaction in place of the old one. because the block data is different, this requires redoing the proof of work. furthermore, the attacker's new version of block 270000 has a different hash, so the original blocks 270001 to 270005 do not "point" to it; thus, the original chain and the attacker's new chain are completely separate. the rule is that in a fork the longest blockchain (ie. the one backed by the largest quantity of proof of work) is taken to be the truth, and so legitimate miners will work on the 270005 chain while the attacker alone is working on the 270000 chain. in order for the attacker to make his blockchain the longest, he would need to have more computational power than the rest of the network combined in order to catch up (hence, "51% attack"). page 8 ethereum.org merkle trees left: it suffices to present only a small number of nodes in a merkle tree to give a proof of the validity of a branch. right: any attempt to change any part of the merkle tree will eventually lead to an inconsistency somewhere up the chain. an important scalability feature of bitcoin is that the block is stored in a multi-level data structure. the "hash" of a block is actually only the hash of the block header, a roughly 200-byte piece of data that contains the timestamp, nonce, previous block hash and the root hash of a data structure called the merkle tree storing all transactions in the block. a merkle tree is a type of binary tree, composed of a set of nodes with a large number of leaf nodes at the bottom of the tree containing the underlying data, a set of intermediate nodes where each node is the hash of its two children, and finally a single root node, also formed from the hash of its two children, representing the "top" of the tree. the purpose of the merkle tree is to allow the data in a block to be delivered piecemeal: a node can download only the header of a block from one source, the small part of the tree relevant to them from page 9 ethereum.org another source, and still be assured that all of the data is correct. the reason why this works is that hashes propagate upward: if a malicious user attempts to swap in a fake transaction into the bottom of a merkle tree, this change will cause a change in the node above, and then a change in the node above that, finally changing the root of the tree and therefore the hash of the block, causing the protocol to register it as a completely different block (almost certainly with an invalid proof of work). the merkle tree protocol is arguably essential to long-term sustainability. a "full node" in the bitcoin network, one that stores and processes the entirety of every block, takes up about 15 gb of disk space in the bitcoin network as of april 2014, and is growing by over a gigabyte per month. currently, this is viable for some desktop computers and not phones, and later on in the future only businesses and hobbyists will be able to participate. a protocol known as "simplified payment verification" (spv) allows for another class of nodes to exist, called "light nodes", which download the block headers, verify the proof of work on the block headers, and then download only the "branches" associated with transactions that are relevant to them. this allows light nodes to determine with a strong guarantee of security what the status of any bitcoin transaction, and their current balance, is while downloading only a very small portion of the entire blockchain. alternative blockchain applications the idea of taking the underlying blockchain idea and applying it to other concepts also has a long history. in 2005, nick szabo came out with the concept of "secure property titles with owner authority", a document describing how "new advances in replicated database technology" will allow for a blockchain-based system for storing a registry of who owns what land, creating an elaborate framework including concepts such as homesteading, adverse possession and georgian land tax. however, there was unfortunately no effective replicated database system available at the time, and so the protocol was never implemented in practice. after 2009, however, once bitcoin's decentralized consensus was developed a number of alternative applications rapidly began to emerge: ● namecoin created in 2010, namecoin is best described as a decentralized name registration database. in decentralized protocols like tor, bitcoin and bitmessage, there needs to be some way of identifying accounts so that other people can interact with them, but in all existing solutions the only kind of identifier available is a pseudorandom hash like1lw79wp5zbqahw1jl5tcibcrhqythaguwy. ideally, one would like to be able to have an account with a name like "george". however, the problem is that if one person can create an account named "george" then someone else can use the same process to register "george" for themselves as well and impersonate them. the only solution is a first-to-file paradigm, where the first registrant succeeds and the second fails a problem perfectly suited for the bitcoin consensus protocol. namecoin is the oldest, and most successful, implementation of a name registration system using such an idea. ● colored coins the purpose of colored coins is to serve as a protocol to allow people to create their own digital currencies or, in the important trivial case of a currency with one unit, digital tokens, page 10 ethereum.org http://www.google.com/url?q=http%3a%2f%2fszabo.best.vwh.net%2fsecuretitle.html&sa=d&sntz=1&usg=afqjcnhuvy9y6t8ve-5eautfyelzkka1bg https://www.google.com/url?q=https%3a%2f%2fnamecoin.org%2f&sa=d&sntz=1&usg=afqjcngeuq4ve088yvm6tqqnlhfqhjol6w https://docs.google.com/a/buterin.com/document/d/1ankp_cvztcmlizw4dvsw6m8q2jc0lizrtluowu2z1be/edit on the bitcoin blockchain. in the colored coins protocol, one "issues" a new currency by publicly assigning a color to a specific bitcoin utxo, and the protocol recursively defines the color of other utxo to be the same as the color of the inputs that the transaction creating them spent (some special rules apply in the case of mixed-color inputs). this allows users to maintain wallets containing only utxo of a specific color and send them around much like regular bitcoins, backtracking through the blockchain to determine the color of any utxo that they receive. ● metacoins the idea behind a metacoin is to have a protocol that lives on top of bitcoin, using bitcoin transactions to store metacoin transactions but having a different state transition function, apply'. because the metacoin protocol cannot prevent invalid metacoin transactions from appearing in the bitcoin blockchain, a rule is added that if apply'(s,tx) returns an error, the protocol defaults to apply'(s,tx) = s. this provides an easy mechanism for creating an arbitrary cryptocurrency protocol, potentially with advanced features that cannot be implemented inside of bitcoin itself, but with a very low development cost since the complexities of mining and networking are already handled by the bitcoin protocol. thus, in general, there are two approaches toward building a consensus protocol: building an independent network, and building a protocol on top of bitcoin. the former approach, while reasonably successful in the case of applications like namecoin, is difficult to implement; each individual implementation needs to bootstrap an independent blockchain, as well as building and testing all of the necessary state transition and networking code. additionally, we predict that the set of applications for decentralized consensus technology will follow a power law distribution where the vast majority of applications would be too small to warrant their own blockchain, and we note that there exist large classes of decentralized applications, particularly decentralized autonomous organizations, that need to interact with each other. the bitcoin-based approach, on the other hand, has the flaw that it does not inherit the simplified payment verification features of bitcoin. spv works for bitcoin because it can use blockchain depth as a proxy for validity; at some point, once the ancestors of a transaction go far enough back, it is safe to say that they were legitimately part of the state. blockchain-based meta-protocols, on the other hand, cannot force the blockchain not to include transactions that are not valid within the context of their own protocols. hence, a fully secure spv meta-protocol implementation would need to backward scan all the way to the beginning of the bitcoin blockchain to determine whether or not certain transactions are valid. currently, all "light" implementations of bitcoin-based meta-protocols rely on a trusted server to provide the data, arguably a highly suboptimal result especially when one of the primary purposes of a cryptocurrency is to eliminate the need for trust. scripting even without any extensions, the bitcoin protocol actually does facilitate a weak version of a concept of "smart contracts". utxo in bitcoin can be owned not just by a public key, but also by a more complicated script expressed in a simple stack-based programming language. in this paradigm, a transaction spending that utxo must provide data that satisfies the script. indeed, even the basic public key ownership mechanism is page 11 ethereum.org implemented via a script: the script takes an elliptic curve signature as input, verifies it against the transaction and the address that owns the utxo, and returns 1 if the verification is successful and 0 otherwise. other, more complicated, scripts exist for various additional use cases. for example, one can construct a script that requires signatures from two out of a given three private keys to validate ("multisig"), a setup useful for corporate accounts, secure savings accounts and some merchant escrow situations. scripts can also be used to pay bounties for solutions to computational problems, and one can even construct a script that says something like "this bitcoin utxo is yours if you can provide an spv proof that you sent a dogecoin transaction of this denomination to me", essentially allowing decentralized cross-cryptocurrency exchange. however, the scripting language as implemented in bitcoin has several important limitations: ● lack of turing-completeness that is to say, while there is a large subset of computation that the bitcoin scripting language supports, it does not nearly support everything. the main category that is missing is loops. this is done to avoid infinite loops during transaction verification; theoretically it is a surmountable obstacle for script programmers, since any loop can be simulated by simply repeating the underlying code many times with an if statement, but it does lead to scripts that are very space-inefficient. for example, implementing an alternative elliptic curve signature algorithm would likely require 256 repeated multiplication rounds all individually included in the code. ● value-blindness there is no way for a utxo script to provide fine-grained control over the amount that can be withdrawn. for example, one powerful use case of an oracle contract would be a hedging contract, where a and b put in $1000 worth of btc and after 30 days the script sends $1000 worth of btc to a and the rest to b. this would require an oracle to determine the value of 1 btc in usd, but even then it is a massive improvement in terms of trust and infrastructure requirement over the fully centralized solutions that are available now. however, because utxo are all-or-nothing, the only way to achieve this is through the very inefficient hack of having many utxo of varying denominations (eg. one utxo of 2k for every k up to 30) and having the oracle pick which utxo to send to a and which to b. ● lack of state utxo can either be spent or unspent; there is no opportunity for multi-stage contracts or scripts which keep any other internal state beyond that. this makes it hard to make multi-stage options contracts, decentralized exchange offers or two-stage cryptographic commitment protocols (necessary for secure computational bounties). it also means that utxo can only be used to build simple, one-off contracts and not more complex "stateful" contracts such as decentralized organizations, and makes meta-protocols difficult to implement. binary state combined with value-blindness also mean that another important application, withdrawal limits, is impossible. ● blockchain-blindness utxo are blind to blockchain data such as the nonce and previous hash. this severely limits applications in gambling, and several other categories, by depriving the scripting language of a potentially valuable source of randomness. page 12 ethereum.org thus, we see three approaches to building advanced applications on top of cryptocurrency: building a new blockchain, using scripting on top of bitcoin, and building a meta-protocol on top of bitcoin. building a new blockchain allows for unlimited freedom in building a feature set, but at the cost of development time and bootstrapping effort. using scripting is easy to implement and standardize, but is very limited in its capabilities, and meta-protocols, while easy, suffer from faults in scalability. with ethereum, we intend to build a generalized framework that can provide the advantages of all three paradigms at the same time. ethereum the intent of ethereum is to merge together and improve upon the concepts of scripting, altcoins and on-chain meta-protocols, and allow developers to create arbitrary consensus-based applications that have the scalability, standardization, feature-completeness, ease of development and interoperability offered by these different paradigms all at the same time. ethereum does this by building what is essentially the ultimate abstract foundational layer: a blockchain with a built-in turing-complete programming language, allowing anyone to write smart contracts and decentralized applications where they can create their own arbitrary rules for ownership, transaction formats and state transition functions. a bare-bones version of namecoin can be written in two lines of code, and other protocols like currencies and reputation systems can be built in under twenty. smart contracts, cryptographic "boxes" that contain value and only unlock it if certain conditions are met, can also be built on top of our platform, with vastly more power than that offered by bitcoin scripting because of the added powers of turing-completeness, value-awareness, blockchain-awareness and state. ethereum accounts in ethereum, the state is made up of objects called "accounts", with each account having a 20-byte address and state transitions being direct transfers of value and information between accounts. an ethereum account contains four fields: ● the nonce, a counter used to make sure each transaction can only be processed once ● the account's current ether balance ● the account's contract code, if present ● the account's storage (empty by default) "ether" is the main internal crypto-fuel of ethereum, and is used to pay transaction fees. in general, there are two types of accounts: externally owned accounts, controlled by private keys, and contract accounts, controlled by their contract code. an externally owned account has no code, and one can send messages from an externally owned account by creating and signing a transaction; in a contract account, every time the page 13 ethereum.org contract account receives a message its code activates, allowing it to read and write to internal storage and send other messages or create contracts in turn. messages and transactions "messages" in ethereum are somewhat similar to “transactions” in bitcoin, but with three important differences. first, an ethereum message can be created either by an external entity or a contract, whereas a bitcoin transaction can only be created externally. second, there is an explicit option for ethereum messages to contain data. finally, the recipient of an ethereum message, if it is a contract account, has the option to return a response; this means that ethereum messages also encompass the concept of functions. the term "transaction" is used in ethereum to refer to the signed data package that stores a message to be sent from an externally owned account. transactions contain the recipient of the message, a signature identifying the sender, the amount of ether and the data to send, as well as two values called startgas and gasprice. in order to prevent exponential blowup and infinite loops in code, each transaction is required to set a limit to how many computational steps of code execution it can spawn, including both the initial message and any additional messages that get spawned during execution. startgas is this limit, and gasprice is the fee to pay to the miner per computational step. if transaction execution "runs out of gas", all state changes revert except for the payment of the fees, and if transaction execution halts with some gas remaining then the remaining portion of the fees is refunded to the sender. there is also a separate transaction type, and corresponding message type, for creating a contract; the address of a contract is calculated based on the hash of the account nonce and transaction data. an important consequence of the message mechanism is the "first class citizen" property of ethereum the idea that contracts have equivalent powers to external accounts, including the ability to send message and create other contracts. this allows contracts to simultaneously serve many different roles: for example, one might have a member of a decentralized organization (a contract) be an escrow account (another contract) between an paranoid individual employing custom quantum-proof lamport signatures (a third contract) and a co-signing entity which itself uses an account with five keys for security (a fourth contract). the strength of the ethereum platform is that the decentralized organization and the escrow contract do not need to care about what kind of account each party to the contract is. page 14 ethereum.org ethereum state transition function the ethereum state transition function, apply(s,tx) -> s' can be defined as follows: 1. check if the transaction is well-formed (ie. has the right number of values), the signature is valid, and the nonce matches the nonce in the sender's account. if not, return an error. 2. calculate the transaction fee as startgas * gasprice, and determine the sending address from the signature. subtract the fee from the sender's account balance and increment the sender's nonce. if there is not enough balance to spend, return an error. 3. initialize gas = startgas, and take off a certain quantity of gas per byte to pay for the bytes in the transaction. 4. transfer the transaction value from the sender's account to the receiving account. if the receiving account does not yet exist, create it. if the receiving account is a contract, run the contract's code either to completion or until the execution runs out of gas. 5. if the value transfer failed because the sender did not have enough money, or the code execution ran out of gas, revert all state changes except the payment of the fees, and add the fees to the miner's account. 6. otherwise, refund the fees for all remaining gas to the sender, and send the fees paid for gas consumed to the miner. page 15 ethereum.org for example, suppose that the contract's code is: if !contract.storage[msg.data[0]]: contract.storage[msg.data[0]] = msg.data[1] note that in reality the contract code is written in the low-level evm code; this example is written in serpent, our high-level language, for clarity, and can be compiled down to evm code. suppose that the contract's storage starts off empty, and a transaction is sent with 10 ether value, 2000 gas, 0.001 ether gasprice, and two data fields: [ 2, 'charlie' ][3]. the process for the state transition function in this case is as follows: 1. check that the transaction is valid and well formed. 2. check that the transaction sender has at least 2000 * 0.001 = 2 ether. if it is, then subtract 2 ether from the sender's account. 3. initialize gas = 2000; assuming the transaction is 170 bytes long and the byte-fee is 5, subtract 850 so that there is 1150 gas left. 4. subtract 10 more ether from the sender's account, and add it to the contract's account. 5. run the code. in this case, this is simple: it checks if the contract's storage at index 2 is used, notices that it is not, and so it sets the storage at index 2 to the value charlie. suppose this takes 187 gas, so the remaining amount of gas is 1150 187 = 963 6. add 963 * 0.001 = 0.963 ether back to the sender's account, and return the resulting state. if there was no contract at the receiving end of the transaction, then the total transaction fee would simply be equal to the provided gasprice multiplied by the length of the transaction in bytes, and the data sent alongside the transaction would be irrelevant. additionally, note that contract-initiated messages can assign a gas limit to the computation that they spawn, and if the sub-computation runs out of gas it gets reverted only to the point of the message call. hence, just like transactions, contracts can secure their limited computational resources by setting strict limits on the sub-computations that they spawn.       page 16 ethereum.org code execution the code in ethereum contracts is written in a low-level, stack-based bytecode language, referred to as "ethereum virtual machine code" or "evm code". the code consists of a series of bytes, where each byte represents an operation. in general, code execution is an infinite loop that consists of repeatedly carrying out the operation at the current program counter (which begins at zero) and then incrementing the program counter by one, until the end of the code is reached or an error or stop or return instruction is detected. the operations have access to three types of space in which to store data: ● the stack, a last-in-first-out container to which 32-byte values can be pushed and popped ● memory, an infinitely expandable byte array ● the contract's long-term storage, a key/value store where keys and values are both 32 bytes. unlike stack and memory, which reset after computation ends, storage persists for the long term. the code can also access the value, sender and data of the incoming message, as well as block header data, and the code can also return a byte array of data as an output. the formal execution model of evm code is surprisingly simple. while the ethereum virtual machine is running, its full computational state can be defined by the tuple (block_state, transaction, message, code, memory, stack, pc, gas), where block_state is the global state containing all accounts and includes balances and storage. every round of execution, the current instruction is found by taking the pc-th byte of code, and each instruction has its own definition in terms of how it affects the tuple. for example, add pops two items off the stack and pushes their sum, reduces gas by 1 and increments pc by 1, and sstore pushes the top two items off the stack and inserts the second item into the contract's storage at the index specified by the first item, as well as reducing gas by up to 200 and incrementing pc by 1. although there are many ways to optimize ethereum via just-in-time compilation, a basic implementation of ethereum can be done in a few hundred lines of code. page 17 ethereum.org blockchain and mining the ethereum blockchain is in many ways similar to the bitcoin blockchain, although it does have some differences. the main difference between ethereum and bitcoin with regard to the blockchain architecture is that, unlike bitcoin, ethereum blocks contain a copy of both the transaction list and the most recent state. aside from that, two other values, the block number and the difficulty, are also stored in the block. the block validation algorithm in ethereum is as follows: 1. check if the previous block referenced exists and is valid. 2. check that the timestamp of the block is greater than that of the referenced previous block and less than 15 minutes into the future 3. check that the block number, difficulty, transaction root, uncle root and gas limit (various low-level ethereum-specific concepts) are valid. 4. check that the proof of work on the block is valid. 5. let s[0] be the state_root of the previous block. 6. let tx be the block's transaction list, with n transactions. for all in in 0...n-1, sets[i+1] = apply(s[i],tx[i]). if any applications returns an error, or if the total gas consumed in the block up until this point exceeds the gaslimit, return an error. 7. let s_final be s[n], but adding the block reward paid to the miner. 8. check if s_final is the same as the state_root. if it is, the block is valid; otherwise, it is not valid. page 18 ethereum.org the approach may seem highly inefficient at first glance, because it needs to store the entire state with each block, but in reality efficiency should be comparable to that of bitcoin. the reason is that the state is stored in the tree structure, and after every block only a small part of the tree needs to be changed. thus, in general, between two adjacent blocks the vast majority of the tree should be the same, and therefore the data can be stored once and referenced twice using pointers (ie. hashes of subtrees). a special kind of tree known as a "patricia tree" is used to accomplish this, including a modification to the merkle tree concept that allows for nodes to be inserted and deleted, and not just changed, efficiently. additionally, because all of the state information is part of the last block, there is no need to store the entire blockchain history a strategy which, if it could be applied to bitcoin, can be calculated to provide 5-20x savings in space. applications in general, there are three types of applications on top of ethereum. the first category is financial applications, providing users with more powerful ways of managing and entering into contracts using their money. this includes sub-currencies, financial derivatives, hedging contracts, savings wallets, wills, and ultimately even some classes of full-scale employment contracts. the second category is semi-financial applications, where money is involved but there is also a heavy non-monetary side to what is being done; a perfect example is self-enforcing bounties for solutions to computational problems. finally, there are applications such as online voting and decentralized governance that are not financial at all. token systems on-blockchain token systems have many applications ranging from sub-currencies representing assets such as usd or gold to company stocks, individual tokens representing smart property, secure unforgeable coupons, and even token systems with no ties to conventional value at all, used as point systems for incentivization. token systems are surprisingly easy to implement in ethereum. the key point to understand is that all a currency, or token systen, fundamentally is is a database with one operation: subtract x units from a and give x units to b, with the proviso that (i) x had at least x units before the transaction and (2) the transaction is approved by a. all that it takes to implement a token system is to implement this logic into a contract. page 19 ethereum.org the basic code for implementing a token system in serpent looks as follows: from = msg.sender  to = msg.data[0]  value = msg.data[1]    if contract.storage[from] >= value:      contract.storage[from] = contract.storage[from] ­ value      contract.storage[to] = contract.storage[to] + value    this is essentially a literal implementation of the "banking system" state transition function described further above in this document. a few extra lines of code need to be added to provide for the initial step of distributing the currency units in the first place and a few other edge cases, and ideally a function would be added to let other contracts query for the balance of an address. but that's all there is to it. theoretically, ethereum-based token systems acting as sub-currencies can potentially include another important feature that on-chain bitcoin-based meta-currencies lack: the ability to pay transaction fees directly in that currency. the way this would be implemented is that the contract would maintain an ether balance with which it would refund ether used to pay fees to the sender, and it would refill this balance by collecting the internal currency units that it takes in fees and reselling them in a constant running auction. users would thus need to "activate" their accounts with ether, but once the ether is there it would be reusable because the contract would refund it each time. financial derivatives and stable-value currencies financial derivatives are the most common application of a "smart contract", and one of the simplest to implement in code. the main challenge in implementing financial contracts is that the majority of them require reference to an external price ticker; for example, a very desirable application is a smart contract that hedges against the volatility of ether (or another cryptocurrency) with respect to the us dollar, but doing this requires the contract to know what the value of eth/usd is. the simplest way to do this is through a "data feed" contract maintained by a specific party (eg. nasdaq) designed so that that party has the ability to update the contract as needed, and providing an interface that allows other contracts to send a message to that contract and get back a response that provides the price. given that critical ingredient, the hedging contract would look as follows: 1. wait for party a to input 1000 ether. 2. wait for party b to input 1000 ether. 3. record the usd value of 1000 ether, calculated by querying the data feed contract, in storage, say this is $x. 4. after 30 days, allow a or b to "ping" the contract in order to send $x worth of ether (calculated by querying the data feed contract again to get the new price) to a and the rest to b. page 20 ethereum.org such a contract would have significant potential in crypto-commerce. one of the main problems cited about cryptocurrency is the fact that it's volatile; although many users and merchants may want the security and convenience of dealing with cryptographic assets, they many not wish to face that prospect of losing 23% of the value of their funds in a single day. up until now, the most commonly proposed solution has been issuer-backed assets; the idea is that an issuer creates a sub-currency in which they have the right to issue and revoke units, and provide one unit of the currency to anyone who provides them (offline) with one unit of a specified underlying asset (eg. gold, usd). the issuer then promises to provide one unit of the underlying asset to anyone who sends back one unit of the crypto-asset. this mechanism allows any non-cryptographic asset to be "uplifted" into a cryptographic asset, provided that the issuer can be trusted. in practice, however, issuers are not always trustworthy, and in some cases the banking infrastructure is too weak, or too hostile, for such services to exist. financial derivatives provide an alternative. here, instead of a single issuer providing the funds to back up an asset, a decentralized market of speculators, betting that the price of a cryptographic reference asset will go up, plays that role. unlike issuers, speculators have no option to default on their side of the bargain because the hedging contract holds their funds in escrow. note that this approach is not fully decentralized, because a trusted source is still needed to provide the price ticker, although arguably even still this is a massive improvement in terms of reducing infrastructure requirements (unlike being an issuer, issuing a price feed requires no licenses and can likely be categorized as free speech) and reducing the potential for fraud.                 page 21 ethereum.org identity and reputation systems the earliest alternative cryptocurrency of all, namecoin, attempted to use a bitcoin-like blockchain to provide a name registration system, where users can register their names in a public database alongside other data. the major cited use case is for a dns system, mapping domain names like "bitcoin.org" (or, in namecoin's case, "bitcoin.bit") to an ip address. other use cases include email authentication and potentially more advanced reputation systems. here is the basic contract to provide a namecoin-like name registration system on ethereum: if !contract.storage[tx.data[0]]:      contract.storage[tx.data[0]] = tx.data[1] the contract is very simple; all it is is a database inside the ethereum network that can be added to, but not modified or removed from. anyone can register a name with some value, and that registration then sticks forever. a more sophisticated name registration contract will also have a "function clause" allowing other contracts to query it, as well as a mechanism for the "owner" (ie. the first registerer) of a name to change the data or transfer ownership. one can even add reputation and web-of-trust functionality on top.   decentralized file storage over the past few years, there have emerged a number of popular online file storage startups, the most prominent being dropbox, seeking to allow users to upload a backup of their hard drive and have the service store the backup and allow the user to access it in exchange for a monthly fee. however, at this point the file storage market is at times relatively inefficient; a cursory look at various existing solutions shows that, particularly at the "uncanny valley" 20-200 gb level at which neither free quotas nor enterprise-level discounts kick in, monthly prices for mainstream file storage costs are such that you are paying for more than the cost of the entire hard drive in a single month. ethereum contracts can allow for the development of a decentralized file storage ecosystem, where individual users can earn small quantities of money by renting out their own hard drives and unused space can be used to further drive down the costs of file storage. the key underpinning piece of such a device would be what we have termed the "decentralized dropbox contract". this contract works as follows. first, one splits the desired data up into blocks, encrypting each block for privacy, and builds a merkle tree out of it. one then makes a contract with the rule that, every n blocks, the contract would pick a random index in the merkle tree (using the previous block hash, accessible from contract code, as a source of randomness), and give x ether to the first entity to supply a transaction with a page 22 ethereum.org http://www.google.com/url?q=http%3a%2f%2fnamecoin.org%2f&sa=d&sntz=1&usg=afqjcnekxddjmwtgniyrxpju5zib-nagiw http://www.google.com/url?q=http%3a%2f%2fen.wikipedia.org%2fwiki%2fdomain_name_system&sa=d&sntz=1&usg=afqjcnevoahupwtvkl4gqzctxn5yi71wiq http://www.google.com/url?q=http%3a%2f%2fonline-storage-service-review.toptenreviews.com%2f&sa=d&sntz=1&usg=afqjcnhvedi9u66mri0ayzjatvrnuc40gg simplified payment verification-like proof of ownership of the block at that particular index in the tree. when a user wants to re-download their file, they can use a micropayment channel protocol (eg. pay 1 szabo per 32 kilobytes) to recover the file; the most fee-efficient approach is for the payer not to publish the transaction until the end, instead replacing the transaction with a slightly more lucrative one with the same nonce after every 32 kilobytes. an important feature of the protocol is that, although it may seem like one is trusting many random nodes not to decide to forget the file, one can reduce that risk down to near-zero by splitting the file into many pieces via secret sharing, and watching the contracts to see each piece is still in some node's possession. if a contract is still paying out money, that provides a cryptographic proof that someone out there is still storing the file.   decentralized autonomous organizations the general concept of a "decentralized organization" is that of a virtual entity that has a certain set of members or shareholders which, perhaps with a 67% majority, have the right to spend the entity's funds and modify its code. the members would collectively decide on how the organization should allocate its funds. methods for allocating a dao's funds could range from bounties, salaries to even more exotic mechanisms such as an internal currency to reward work. this essentially replicates the legal trappings of a traditional company or nonprofit but using only cryptographic blockchain technology for enforcement. so far much of the talk around daos has been around the "capitalist" model of a "decentralized autonomous corporation" (dac) with dividend-receiving shareholders and tradable shares; an alternative, perhaps described as a "decentralized autonomous community", would have all members have an equal share in the decision making and require 67% of existing members to agree to add or remove a member. the requirement that one person can only have one membership would then need to be enforced collectively by the group. a general outline for how to code a do is as follows. the simplest design is simply a piece of self-modifying code that changes if two thirds of members agree on a change. although code is theoretically immutable, one can easily get around this and have de-facto mutability by having chunks of the code in separate contracts, and having the address of which contracts to call stored in the modifiable storage. in a simple implementation of such a dao contract, there would be three transaction types, distinguished by the data provided in the transaction: ● [0,i,k,v]to register a proposal with index i to change the address at storage index k to value v ● [0,i] to register a vote in favor of proposal i ● [2,i] to finalize proposal i if enough votes have been made the contract would then have clauses for each of these. it would maintain a record of all open storage changes, along with a list of who voted for them. it would also have a list of all members. when any storage page 23 ethereum.org change gets to two thirds of members voting for it, a finalizing transaction could execute the change. a more sophisticated skeleton would also have built-in voting ability for features like sending a transaction, adding members and removing members, and may even provide for liquid democracy-style vote delegation (ie. anyone can assign someone to vote for them, and assignment is transitive so if a assigns b and b assigns c then c determines a's vote). this design would allow the do to grow organically as a decentralized community, allowing people to eventually delegate the task of filtering out who is a member to specialists, although unlike in the "current system" specialists can easily pop in and out of existence over time as individual community members change their alignments. an alternative model is for a decentralized corporation, where any account can have zero or more shares, and two thirds of the shares are required to make a decision. a complete skeleton would involve asset management functionality, the ability to make an offer to buy or sell shares, and the ability to accept offers (preferably with an order-matching mechanism inside the contract). delegation would also exist liquid democracy-style, generalizing the concept of a "board of directors". in the future, more advanced mechanisms for organizational governance may be implemented; it is at this point that a decentralized organization (do) can start to be described as a decentralized autonomous organization (dao). the difference between a do and a dao is fuzzy, but the general dividing line is whether the governance is generally carried out via a political-like process or an “automatic” process; a good intuitive test is the “no common language” criterion: can the organization still function if no two members spoke the same language? clearly, a simple traditional shareholder-style corporation would fail, whereas something like the bitcoin protocol would be much more likely to succeed. robin hanson’s futarchy, a mechanism for organizational governance via prediction markets, is a good example of what truly “autonomous” governance might look like. note that one should not necessarily assume that all daos are superior to all dos; automation is simply a paradigm that is likely to have have very large benefits in certain particular places and may not be practical in others, and many semi-daos are also likely to exist. further applications 1. savings wallets. suppose that alice wants to keep her funds safe, but is worried that she will lose or someone will hack her private key. she puts ether into a contract with bob, a bank, as follows: ● alice alone can withdraw a maximum of 1% of the funds per day. ● bob alone can withdraw a maximum of 1% of the funds per day, but alice has the ability to make a transaction with her key shutting off this ability. ● alice and bob together can withdraw anything. normally, 1% per day is enough for alice, and if alice wants to withdraw more she can contact bob for help. if alice's key gets hacked, she runs to bob to move the funds to a new contract. if she loses her key, bob will get the funds out eventually. if bob turns out to be malicious, then she can turn off his ability to withdraw. page 24 ethereum.org http://www.google.com/url?q=http%3a%2f%2fen.wikipedia.org%2fwiki%2fdelegative_democracy&sa=d&sntz=1&usg=afqjcnghacp1r99dcd2y2sf-bwkkqbezkq 2. crop insurance. one can easily make a financial derivatives contract but using a data feed of the weather instead of any price index. if a farmer in iowa purchases a derivative that pays out inversely based on the precipitation in iowa, then if there is a drought, the farmer will automatically receive money and if there is enough rain the farmer will be happy because their crops would do well. 3. a decentralized data feed. for financial contracts for difference, it may actually be possible to decentralize the data feed via a protocol called "schellingcoin". schellingcoin basically works as follows: n parties all put into the system the value of a given datum (eg. the eth/usd price), the values are sorted, and everyone between the 25th and 75th percentile gets one token as a reward. everyone has the incentive to provide the answer that everyone else will provide, and the only value that a large number of players can realistically agree on is the obvious default: the truth. this creates a decentralized protocol that can theoretically provide any number of values, including the eth/usd price, the temperature in berlin or even the result of a particular hard computation. 4. smart multi-signature escrow. bitcoin allows multisignature transaction contracts where, for example, three out of a given five keys can spend the funds. ethereum allows for more granularity; for example, four out of five can spend everything, three out of five can spend up to 10% per day, and two out of five can spend up to 0.5% per day. additionally, ethereum multisig is asynchronous two parties can register their signatures on the blockchain at different times and the last signature will automatically send the transaction. 5. cloud computing. the evm technology can also be used to create a verifiable computing environment, allowing users to ask others to carry out computations and then optionally ask for proofs that computations at certain randomly selected checkpoints were done correctly. this allows for the creation of a cloud computing market where any user can participate with their desktop, laptop or specialized server, and spot-checking together with security deposits can be used to ensure that the system is trustworthy (ie. nodes cannot profitably cheat). although such a system may not be suitable for all tasks; tasks that require a high level of inter-process communication, for example, cannot easily be done on a large cloud of nodes. other tasks, however, are much easier to parallelize; projects like seti@home, folding@home and genetic algorithms can easily be implemented on top of such a platform. 6. peer-to-peer gambling. any number of peer-to-peer gambling protocols, such as frank stajano and richard clayton's cyberdice, can be implemented on the ethereum blockchain. the simplest gambling protocol is actually simply a contract for difference on the next block hash, and more advanced protocols can be built up from there, creating gambling services with near-zero fees that have no ability to cheat. 7. prediction markets. provided an oracle or schellingcoin, prediction markets are also easy to implement, and prediction markets together with schellingcoin may prove to be the first mainstream application of futarchy as a governance protocol for decentralized organizations. 8. on-chain decentralized marketplaces, using the identity and reputation system as a base. page 25 ethereum.org http://www.google.com/url?q=http%3a%2f%2fblog.ethereum.org%2f2014%2f03%2f28%2fschellingcoin-a-minimal-trust-universal-data-feed%2f&sa=d&sntz=1&usg=afqjcneu0_xce1ultcmndljsc9xyip96cq http://www.google.com/url?q=http%3a%2f%2fwww.cl.cam.ac.uk%2f%257efms27%2fpapers%2f2008-stajanocla-cyberdice.pdf&sa=d&sntz=1&usg=afqjcne0-foadvtzjxtgwxmitcf363zkww http://www.google.com/url?q=http%3a%2f%2fhanson.gmu.edu%2ffutarchy.html&sa=d&sntz=1&usg=afqjcnh-mxj9s_pcwjyedffj_gk0zoajsa   miscellanea and concerns modified ghost implementation the "greedy heaviest observed subtree" (ghost) protocol is an innovation first introduced by yonatan sompolinsky and aviv zohar in december 2013. the motivation behind ghost is that blockchains with fast confirmation times currently suffer from reduced security due to a high stale rate because blocks take a certain time to propagate through the network, if miner a mines a block and then miner b happens to mine another block before miner a's block propagates to b, miner b's block will end up wasted and will not contribute to network security. furthermore, there is a centralization issue: if miner a is a mining pool with 30% hashpower and b has 10% hashpower, a will have a risk of producing a stale block 70% of the time (since the other 30% of the time a produced the last block and so will get mining data immediately) whereas b will have a risk of producing a stale block 90% of the time. thus, if the block interval is short enough for the stale rate to be high, a will be substantially more efficient simply by virtue of its size. with these two effects combined, blockchains which produce blocks quickly are very likely to lead to one mining pool having a large enough percentage of the network hashpower to have de facto control over the mining process. as described by sompolinsky and zohar, ghost solves the first issue of network security loss by including stale blocks in the calculation of which chain is the "longest"; that is to say, not just the parent and further ancestors of a block, but also the stale children of the block's ancestors (in ethereum jargon, "uncles") are added to the calculation of which block has the largest total proof of work backing it. to solve the second issue of centralization bias, we go beyond the protocol described by sompolinsky and zohar, and also allow stales to be registered into the main chain to receive a block reward: a stale block receives 93.75% of its base reward, and the nephew that includes the stale block receives the remaining 6.25%. transaction fees, however, are not awarded to uncles. ethereum implements a simplified version of ghost which only goes down five levels. specifically, a stale block can only be included as an uncle by the 2nd to 5th generation child of its parent, and not any block with a more distant relation (eg. 6th generation child of a parent, or 3rd generation child of a grandparent). this was done for several reasons. first, unlimited ghost would include too many complications into the calculation of which uncles for a given block are valid. second, unlimited ghost with compensation as used in ethereum removes the incentive for a miner to mine on the main chain and not the chain of a public attacker. finally, calculations show that five-level ghost with incentivization is over 95% efficient even with a 15s block time, and miners with 25% hashpower show centralization gains of less than 3%. page 26 ethereum.org http://www.google.com/url?q=http%3a%2f%2fwww.cs.huji.ac.il%2f%257eavivz%2fpubs%2f13%2fbtc_scalability_full.pdf&sa=d&sntz=1&usg=afqjcne-lgxsnn5wcivnujz_fd1lx8gesg   fees because every transaction published into the blockchain imposes on the network the cost of needing to download and verify it, there is a need for some regulatory mechanism, typically involving transaction fees, to prevent abuse. the default approach, used in bitcoin, is to have purely voluntary fees, relying on miners to act as the gatekeepers and set dynamic minimums. this approach has been received very favorably in the bitcoin community particularly because it is "market-based", allowing supply and demand between miners and transaction senders determine the price. the problem with this line of reasoning is, however, that transaction processing is not a market; although it is intuitively attractive to construe transaction processing as a service that the miner is offering to the sender, in reality every transaction that a miner includes will need to be processed by every node in the network, so the vast majority of the cost of transaction processing is borne by third parties and not the miner that is making the decision of whether or not to include it. hence, tragedy-of-the-commons problems are very likely to occur. however, as it turns out this flaw in the market-based mechanism, when given a particular inaccurate simplifying assumption, magically cancels itself out. the argument is as follows. suppose that: 1. a transaction leads to k operations, offering the reward kr to any miner that includes it where r is set by the sender and k and r are (roughly) visible to the miner beforehand. 2. an operation has a processing cost of c to any node (ie. all nodes have equal efficiency) 3. there are n mining nodes, each with exactly equal processing power (ie. 1/n of total) 4. no non-mining full nodes exist. a miner would be willing to process a transaction if the expected reward is greater than the cost. thus, the expected reward is kr/n since the miner has a 1/n chance of processing the next block, and the processing cost for the miner is simply kc. hence, miners will include transactions where kr/n > kc, or r > nc. note that r is the per-operation fee provided by the sender, and is thus a lower bound on the benefit that the sender derives from the transaction, and nc is the cost to the entire network together of processing an operation. hence, miners have the incentive to include only those transactions for which the total utilitarian benefit exceeds the cost. however, there are several important deviations from those assumptions in reality: 1. the miner does pay a higher cost to process the transaction than the other verifying nodes, since the extra verification time delays block propagation and thus increases the chance the block will become a stale. 2. there do exist non-mining full nodes. page 27 ethereum.org 3. the mining power distribution may end up radically inegalitarian in practice. 4. speculators, political enemies and crazies whose utility function includes causing harm to the network do exist, and they can cleverly set up contracts whose cost is much lower than the cost paid by other verifying nodes. point 1 above provides a tendency for the miner to include fewer transactions, and point 2 increases nc; hence, these two effects at least partially cancel each other out. points 3 and 4 are the major issue; to solve them we simply institute a floating cap: no block can have more operations than blk_limit_factor times the long-term exponential moving average. specifically: blk.oplimit = floor((blk.parent.oplimit * (emafactor 1) + floor(parent.opcount * blk_limit_factor)) / ema_factor) blk_limit_factor and ema_factor are constants that will be set to 65536 and 1.5 for the time being, but will likely be changed after further analysis. computation and turing-completeness an important note is that the ethereum virtual machine is turing-complete; this means that evm code can encode any computation that can be conceivably carried out, including infinite loops. evm code allows looping in two ways. first, there is a jump instruction that allows the program to jump back to a previous spot in the code, and a jumpi instruction to do conditional jumping, allowing for statements like while x < 27: x = x * 2. second, contracts can call other contracts, potentially allowing for looping through recursion. this naturally leads to a problem: can malicious users essentially shut miners and full nodes down by forcing them to enter into an infinite loop? the issue arises because of a problem in computer science known as the halting problem: there is no way to tell, in the general case, whether or not a given program will ever halt. as described in the state transition section, our solution works by requiring a transaction to set a maximum number of computational steps that it is allowed to take, and if execution takes longer computation is reverted but fees are still paid. messages work in the same way. to show the motivation behind our solution, consider the following examples: ● an attacker creates a contract which runs an infinite loop, and then sends a transaction activating that loop to the miner. the miner will process the transaction, running the infinite loop, and wait for it to run out of gas. even though the execution runs out of gas and stops halfway through, the transaction is still valid and the miner still claims the fee from the attacker for each computational step. ● an attacker creates a very long infinite loop with the intent of forcing the miner to keep computing for such a long time that by the time computation finishes a few more blocks will have come out and it will not be possible for the miner to include the transaction to claim the fee. however, page 28 ethereum.org the attacker will be required to submit a value for startgas limiting the number of computational steps that execution can take, so the miner will know ahead of time that the computation will take an excessively large number of steps. ● an attacker sees a contract with code of some form like send(a,contract.storage[a]); contract.storage[a] = 0, and sends a transaction with just enough gas to run the first step but not the second (ie. making a withdrawal but not letting the balance go down). the contract author does not need to worry about protecting against such attacks, because if execution stops halfway through the changes get reverted. ● a financial contract works by taking the median of nine proprietary data feeds in order to minimize risk. an attacker takes over one of the data feeds, which is designed to be modifiable via the variable-address-call mechanism described in the section on daos, and converts it to run an infinite loop, thereby attempting to force any attempts to claim funds from the financial contract to run out of gas. however, the financial contract can set a gas limit on the message to prevent this problem. the alternative to turing-completeness is turing-incompleteness, where jump and jumpi do not exist and only one copy of each contract is allowed to exist in the call stack at any given time. with this system, the fee system described and the uncertainties around the effectiveness of our solution might not be necessary, as the cost of executing a contract would be bounded above by its size. additionally, turing-incompleteness is not even that big a limitation; out of all the contract examples we have conceived internally, so far only one required a loop, and even that loop could be removed by making 26 repetitions of a one-line piece of code. given the serious implications of turing-completeness, and the limited benefit, why not simply have a turing-incomplete language? in reality, however, turing-incompleteness is far from a neat solution to the problem. to see why, consider the following contracts: c0: call(c1); call(c1);  c1: call(c2); call(c2);  c2: call(c3); call(c3);  ...  c49: call(c50); call(c50);  c50: (run one step of a program and record the change in storage) now, send a transaction to a. thus, in 51 transactions, we have a contract that takes up 250 computational steps. miners could try to detect such logic bombs ahead of time by maintaining a value alongside each contract specifying the maximum number of computational steps that it can take, and calculating this for contracts calling other contracts recursively, but that would require miners to forbid contracts that create other contracts (since the creation and execution of all 50 contracts above could easily be rolled into a single contract). another problematic point is that the address field of a message is a variable, so in general it may not even be possible to tell which other contracts a given contract will call ahead of time. hence, all in all, we have a surprising conclusion: turing-completeness is surprisingly easy to manage, and the lack of page 29 ethereum.org turing-completeness is equally surprisingly difficult to manage unless the exact same controls are in place but in that case why not just let the protocol be turing-complete?   currency and issuance the ethereum network includes its own built-in currency, ether, which serves the dual purpose of providing a primary liquidity layer to allow for efficient exchange between various types of digital assets and, more importantly, of providing a mechanism for paying transaction fees. for convenience and to avoid future argument (see the current mbtc/ubtc/satoshi debate in bitcoin), the denominations will be pre-labelled: ● 1: wei ● 10^12: szabo ● 10^15: finney ● 10^18: ether this should be taken as an expanded version of the concept of "dollars" and "cents" or "btc" and "satoshi". in the near future, we expect "ether" to be used for ordinary transactions, "finney" for microtransactions and "szabo" and "wei" for technical discussions around fees and protocol implementation. the issuance model will be as follows: ● ether will be released in a currency sale at the price of 1337-2000 ether per btc, a mechanism intended to fund the ethereum organization and pay for development that has been used with success by a number of other cryptographic platforms. earlier buyers will benefit from larger discounts. the btc received from the sale will be used entirely to pay salaries and bounties to developers, researchers and projects in the cryptocurrency ecosystem. ● 0.099x the total amount sold will be allocated to early contributors who participated in development before btc funding or certainty of funding was available, and another 0.099x will be allocated to long-term research projects. ● 0.26x the total amount sold will be allocated to miners per year forever after that point. page 30 ethereum.org issuance breakdown the permanent linear supply growth model reduces the risk of what some see as excessive wealth concentration in bitcoin, and gives individuals living in present and future eras a fair chance to acquire currency units, while at the same time discouraging depreciation of ether because the "supply growth rate" as a percentage still tends to zero over time. we also theorize that because coins are always lost over time due to carelessness, death, etc, and coin loss can be modeled as a percentage of the total supply per year, that the total currency supply in circulation will in fact eventually stabilize at a value equal to the annual issuance divided by the loss rate (eg. at a loss rate of 1%, once the supply reaches 26x then 0.26x will be mined and 0.26x lost every year, creating an equilibrium).   group at launch after 1 year after 5 years currency units 1.198x 1.458x 2.498x purchasers 83.5% 68.6% 40.0% early contributor distribution 8.26% 6.79% 3.96% long-term endowment 8.26% 6.79% 3.96% miners 0% 17.8% 52.0% despite the linear currency issuance, just like with bitcoin over time the supply growth rate nevertheless tends to zero. page 31 ethereum.org mining centralization the bitcoin mining algorithm basically works by having miners compute sha256 on slightly modified versions of the block header millions of times over and over again, until eventually one node comes up with a version whose hash is less than the target (currently around 2190). however, this mining algorithm is vulnerable to two forms of centralization. first, the mining ecosystem has come to be dominated by asics (application-specific integrated circuits), computer chips designed for, and therefore thousands of times more efficient at, the specific task of bitcoin mining. this means that bitcoin mining is no longer a highly decentralized and egalitarian pursuit, requiring millions of dollars of capital to effectively participate in. second, most bitcoin miners do not actually perform block validation locally; instead, they rely on a centralized mining pool to provide the block headers. this problem is arguably worse: as of the time of this writing, the top two mining pools indirectly control roughly 50% of processing power in the bitcoin network, although this is mitigated by the fact that miners can switch to other mining pools if a pool or coalition attempts a 51% attack. the current intent at ethereum is to use a mining algorithm based on randomly generating a unique hash function for every 1000 nonces, using a sufficiently broad range of computation to remove the benefit of specialized hardware. such a strategy will certainly not reduce the gain of centralization to zero, but it does not need to. note that each individual user, on their private laptop or desktop, can perform a certain quantity of mining activity almost for free, paying only electricity costs, but after the point of 100% cpu utilization of their computer additional mining will require them to pay for both electricity and hardware. asic mining companies need to pay for electricity and hardware starting from the first hash. hence, if the centralization gain can be kept to below this ratio, (e + h) / e, then even if asics are made there will still be room for ordinary miners. additionally, we intend to design the mining algorithm so that mining requires access to the entire blockchain, forcing miners to store the entire blockchain and at least be capable of verifying every transaction. this removes the need for centralized mining pools; although mining pools can still serve the legitimate role of evening out the randomness of reward distribution, this function can be served equally well by peer-to-peer pools with no central control. it additionally helps fight centralization, by increasing the number of full nodes in the network so that the network remains reasonably decentralized even if most ordinary users prefer light clients.   page 32 ethereum.org scalability one common concern about ethereum is the issue of scalability. like bitcoin, ethereum suffers from the flaw that every transaction needs to be processed by every node in the network. with bitcoin, the size of the current blockchain rests at about 20 gb, growing by about 1 mb per hour. if the bitcoin network were to process visa's 2000 transactions per second, it would grow by 1 mb per three seconds (1 gb per hour, 8 tb per year). ethereum is likely to suffer a similar growth pattern, worsened by the fact that there will be many applications on top of the ethereum blockchain instead of just a currency as is the case with bitcoin, but ameliorated by the fact that ethereum full nodes need to store just the state instead of the entire blockchain history. the problem with such a large blockchain size is centralization risk. if the blockchain size increases to, say, 100 tb, then the likely scenario would be that only a very small number of large businesses would run full nodes, with all regular users using light spv nodes. in such a situation, there arises the potential concern that the full nodes could band together and all agree to cheat in some profitable fashion (eg. change the block reward, give themselves btc). light nodes would have no way of detecting this immediately. of course, at least one honest full node would likely exist, and after a few hours information about the fraud would trickle out through channels like reddit, but at that point it would be too late: it would be up to the ordinary users to organize an effort to blacklist the given blocks, a massive and likely infeasible coordination problem on a similar scale as that of pulling off a successful 51% attack. in the case of bitcoin, this is currently a problem, but there exists a blockchain modification suggested by peter todd which will alleviate this issue. in the near term, ethereum will use two additional strategies to cope with this problem. first, because of the blockchain-based mining algorithms, at least every miner will be forced to be a full node, creating a lower bound on the number of full nodes. second and more importantly, however, we will include an intermediate state tree root in the blockchain after processing each transaction. even if block validation is centralized, as long as one honest verifying node exists, the centralization problem can be circumvented via a verification protocol. if a miner publishes an invalid block, that block must either be badly formatted, or the state s[n] is incorrect. since s[0] is known to be correct, there must be some first state s[i] that is incorrect where s[i-1] is correct. the verifying node would provide the index i, along with a "proof of invalidity" consisting of the subset of patricia tree nodes needing to process apply(s[i-1],tx[i]) -> s[i]. nodes would be able to use those nodes to run that part of the computation, and see that the s[i] generated does not match the s[i] provided. another, more sophisticated, attack would involve the malicious miners publishing incomplete blocks, so the full information does not even exist to determine whether or not blocks are valid. the solution to this is a challenge-response protocol: verification nodes issue "challenges" in the form of target transaction indices, and upon receiving a node a light node treats the block as untrusted until another node, whether the miner or another verifier, provides a subset of patricia nodes as a proof of validity. page 33 ethereum.org http://www.google.com/url?q=http%3a%2f%2fsourceforge.net%2fp%2fbitcoin%2fmailman%2fmessage%2f31709140%2f&sa=d&sntz=1&usg=afqjcnhbbgsohkhjtsrpkpa55-32-knqcg putting it all together: decentralized applications the contract mechanism described above allows anyone to build what is essentially a command line application run on a virtual machine that is executed by consensus across the entire network, allowing it to modify a globally accessible state as its “hard drive”. however, for most people, the command line interface that is the transaction sending mechanism is not sufficiently user-friendly to make decentralization an attractive mainstream alternative. to this end, a complete “decentralized application” should consist of both low-level business-logic components, whether implemented entirely on ethereum, using a combination of ethereum and other systems (eg. a p2p messaging layer, one of which is currently planned to be put into the ethereum clients) or other systems entirely, and high-level graphical user interface components. the ethereum client’s design is to serve as a web browser, but include support for a “eth” javascript api object, which specialized web pages viewed in the client will be able to use to interact with the ethereum blockchain. from the point of view of the “traditional” web, these web pages are entirely static content, since the blockchain and other decentralized protocols will serve as a complete replacement for the server for the purpose of handling user-initiated requests. eventually, decentralized protocols, hopefully themselves in some fashion using ethereum, may be used to store the web pages themselves. conclusion the ethereum protocol was originally conceived as an upgraded version of a cryptocurrency, providing advanced features such as on-blockchain escrow, withdrawal limits and financial contracts, gambling markets and the like via a highly generalized programming language. the ethereum protocol would not "support" any of the applications directly, but the existence of a turing-complete programming language means that arbitrary contracts can theoretically be created for any transaction type or application. what is more interesting about ethereum, however, is that the ethereum protocol moves far beyond just currency. protocols and decentralized applications around decentralized file storage, decentralized computation and decentralized prediction markets, among dozens of other such concepts, have the potential to substantially increase the efficiency of the computational industry, and provide a massive boost to other peer-to-peer protocols by adding for the first time an economic layer. finally, there is also a substantial array of applications that have nothing to do with money at all. the concept of an arbitrary state transition function as implemented by the ethereum protocol provides for a platform with unique potential; rather than being a closed-ended, single-purpose protocol intended for a specific array of applications in data storage, gambling or finance, ethereum is open-ended by design, and we believe that it is extremely well-suited to serving as a foundational layer for a very large number of both financial and non-financial protocols in the years to come. page 34 ethereum.org         notes and further reading notes 1. a sophisticated reader may notice that in fact a bitcoin address is the hash of the elliptic curve public key, and not the public key itself. however, it is in fact perfectly legitimate cryptographic terminology to refer to the pubkey hash as a public key itself. this is because bitcoin's cryptography can be considered to be a custom digital signature algorithm, where the public key consists of the hash of the ecc pubkey, the signature consists of the ecc pubkey concatenated with the ecc signature, and the verification algorithm involves checking the ecc pubkey in the signature against the ecc pubkey hash provided as a public key and then verifying the ecc signature against the ecc pubkey. 2. technically, the median of the 11 previous blocks. 3. internally, 2 and "charlie" are both numbers, with the latter being in big-endian base 256 representation. numbers can be at least 0 and at most 2^256-1. further reading 1. intrinsic value: https://tinyurl.com/bitcoinmag-intrinsicvalue 2. smart property: https://en.bitcoin.it/wiki/smart_property 3. smart contracts: https://en.bitcoin.it/wiki/contracts 4. b-money: http://www.weidai.com/bmoney.txt 5. reusable proofs of work: http://www.finney.org/~hal/rpow/ 6. secure property titles with owner authority: http://szabo.best.vwh.net/securetitle.html 7. bitcoin whitepaper: http://bitcoin.org/bitcoin.pdf 8. namecoin: https://namecoin.org/ 9. zooko's triangle: http://en.wikipedia.org/wiki/zooko's_triangle 10. colored coins whitepaper: https://tinyurl.com/coloredcoin-whitepaper 11. mastercoin whitepaper: https://github.com/mastercoin-msc/spec 12. decentralized autonomous corporations, bitcoin magazine: https://tinyurl.com/bootstrapping-dacs 13. simplified payment verification:https://en.bitcoin.it/wiki/scalability#simplifiedpaymentverification 14. merkle trees: http://en.wikipedia.org/wiki/merkle_tree 15. patricia trees: http://en.wikipedia.org/wiki/patricia_tree 16. ghost: http://www.cs.huji.ac.il/~avivz/pubs/13/btc_scalability_full.pdf 17. storj and autonomous agents, jeff garzik: https://tinyurl.com/storj-agents 18. mike hearn on smart property at turing festival: http://www.youtube.com/watch?v=pu4pamfpo5y page 35 ethereum.org http://www.google.com/url?q=http%3a%2f%2fbitcoinmagazine.com%2f8640%2fan-exploration-of-intrinsic-value-what-it-is-why-bitcoin-doesnt-have-it-and-why-bitcoin-does-have-it%2f&sa=d&sntz=1&usg=afqjcnhihj5joz_hutamkn106c0c2k7t7w https://www.google.com/url?q=https%3a%2f%2fen.bitcoin.it%2fwiki%2fsmart_property&sa=d&sntz=1&usg=afqjcnelwkf15lophty2dnlvkzibh2sdww https://www.google.com/url?q=https%3a%2f%2fen.bitcoin.it%2fwiki%2fcontracts&sa=d&sntz=1&usg=afqjcnfhamrwxsdixvlygdn0cgpervuhtq http://www.google.com/url?q=http%3a%2f%2fwww.weidai.com%2fbmoney.txt&sa=d&sntz=1&usg=afqjcneythbdnq0vi1g--o4hwyuoscbiqa http://www.google.com/url?q=http%3a%2f%2fwww.finney.org%2f%257ehal%2frpow%2f&sa=d&sntz=1&usg=afqjcnfzg0g7mwxclfki1yqvmfqeeipe2a http://www.google.com/url?q=http%3a%2f%2fszabo.best.vwh.net%2fsecuretitle.html&sa=d&sntz=1&usg=afqjcnhuvy9y6t8ve-5eautfyelzkka1bg http://www.google.com/url?q=http%3a%2f%2fbitcoin.org%2fbitcoin.pdf&sa=d&sntz=1&usg=afqjcnh4hwdyo83i-rg9o6mycy3pr7cydq https://www.google.com/url?q=https%3a%2f%2fnamecoin.org%2f&sa=d&sntz=1&usg=afqjcngeuq4ve088yvm6tqqnlhfqhjol6w http://www.google.com/url?q=http%3a%2f%2fen.wikipedia.org%2fwiki%2fzooko's_triangle&sa=d&sntz=1&usg=afqjcneph_ul1f3snmb6wxep1vyrs7ta9w https://docs.google.com/a/buterin.com/document/d/1ankp_cvztcmlizw4dvsw6m8q2jc0lizrtluowu2z1be/edit https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fmastercoin-msc%2fspec&sa=d&sntz=1&usg=afqjcngm2mijuey8xhk6q_pehy_qgpucdw http://www.google.com/url?q=http%3a%2f%2fbitcoinmagazine.com%2f7050%2fbootstrapping-a-decentralized-autonomous-corporation-part-i%2f&sa=d&sntz=1&usg=afqjcnf8ximt3yrvaid-x0dofldgk1jsjg https://www.google.com/url?q=https%3a%2f%2fen.bitcoin.it%2fwiki%2fscalability%23simplifiedpaymentverification&sa=d&sntz=1&usg=afqjcnesg5bntnvzrkye_vsx0rmd2qj79a http://www.google.com/url?q=http%3a%2f%2fen.wikipedia.org%2fwiki%2fmerkle_tree&sa=d&sntz=1&usg=afqjcnfoxwjlydwfxd5nwrhbkav2lvv8mw http://www.google.com/url?q=http%3a%2f%2fen.wikipedia.org%2fwiki%2fpatricia_tree&sa=d&sntz=1&usg=afqjcnhqp4dk_9jawfuviozo4eewjh3nxg http://www.google.com/url?q=http%3a%2f%2fwww.cs.huji.ac.il%2f%257eavivz%2fpubs%2f13%2fbtc_scalability_full.pdf&sa=d&sntz=1&usg=afqjcne-lgxsnn5wcivnujz_fd1lx8gesg http://garzikrants.blogspot.ca/2013/01/storj-and-bitcoin-autonomous-agents.html http://www.youtube.com/watch?v=pu4pamfpo5y 19. ethereum rlp: https://github.com/ethereum/wiki/wiki/%5benglish%5d-rlp 20. ethereum merkle patricia trees: https://github.com/ethereum/wiki/wiki/%5benglish%5d-patricia-tree 21. peter todd on merkle sum trees:http://sourceforge.net/p/bitcoin/mailman/message/31709140/ page 36 ethereum.org https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-rlp&sa=d&sntz=1&usg=afqjcngea9ti9rk93ndhd11kkkrxymdega https://www.google.com/url?q=https%3a%2f%2fgithub.com%2fethereum%2fwiki%2fwiki%2f%255benglish%255d-patricia-tree&sa=d&sntz=1&usg=afqjcnffj9qkmnjwzbulvlmmwzqulrlu7w http://www.google.com/url?q=http%3a%2f%2fsourceforge.net%2fp%2fbitcoin%2fmailman%2fmessage%2f31709140%2f&sa=d&sntz=1&usg=afqjcnhbbgsohkhjtsrpkpa55-32-knqcg ethereum yellow paper: a formal specification of ethereum, a programmable blockchain ethereum: a secure decentralised generalised transaction ledger shanghai version 9fde3f4 – 2024-09-02 dr. gavin wood founder, ethereum & parity gavin@parity.io abstract. the blockchain paradigm when coupled with cryptographically-secured transactions has demonstrated its utility through a number of projects, with bitcoin being one of the most notable ones. each such project can be seen as a simple application on a decentralised, but singleton, compute resource. we can call this paradigm a transactional singleton machine with shared-state. ethereum implements this paradigm in a generalised manner. furthermore it provides a plurality of such resources, each with a distinct state and operating code but able to interact through a message-passing framework with others. we discuss its design, implementation issues, the opportunities it provides and the future hurdles we envisage. 1. introduction with ubiquitous internet connections in most places of the world, global information transmission has become incredibly cheap. technology-rooted movements like bitcoin have demonstrated through the power of the default, consensus mechanisms, and voluntary respect of the social contract, that it is possible to use the internet to make a decentralised value-transfer system that can be shared across the world and virtually free to use. this system can be said to be a very specialised version of a cryptographically secure, transaction-based state machine. follow-up systems such as namecoin adapted this original “currency application” of the technology into other applications, albeit rather simplistic ones. ethereum is a project which attempts to build the generalised technology; technology on which all transactionbased state machine concepts may be built. moreover it aims to provide to the end-developer a tightly integrated end-to-end system for building software on a hitherto unexplored compute paradigm in the mainstream: a trustful object messaging compute framework. 1.1. driving factors. there are many goals of this project; one key goal is to facilitate transactions between consenting individuals who would otherwise have no means to trust one another. this may be due to geographical separation, interfacing difficulty, or perhaps the incompatibility, incompetence, unwillingness, expense, uncertainty, inconvenience, or corruption of existing legal systems. by specifying a state-change system through a rich and unambiguous language, and furthermore architecting a system such that we can reasonably expect that an agreement will be thus enforced autonomously, we can provide a means to this end. dealings in this proposed system would have several attributes not often found in the real world. the incorruptibility of judgement, often difficult to find, comes naturally from a disinterested algorithmic interpreter. transparency, or being able to see exactly how a state or judgement came about through the transaction log and rules or instructional codes, never happens perfectly in human-based systems since natural language is necessarily vague, information is often lacking, and plain old prejudices are difficult to shake. overall, we wish to provide a system such that users can be guaranteed that no matter with which other individuals, systems or organisations they interact, they can do so with absolute confidence in the possible outcomes and how those outcomes might come about. 1.2. previous work. buterin [2013] first proposed the kernel of this work in late november, 2013. though now evolved in many ways, the key functionality of a blockchain with a turing-complete language and an effectively unlimited inter-transaction storage capability remains unchanged. dwork and naor [1992] provided the first work into the usage of a cryptographic proof of computational expenditure (“proof-of-work”) as a means of transmitting a value signal over the internet. the value-signal was utilised here as a spam deterrence mechanism rather than any kind of currency, but critically demonstrated the potential for a basic data channel to carry a strong economic signal, allowing a receiver to make a physical assertion without having to rely upon trust. back [2002] later produced a system in a similar vein. the first example of utilising the proof-of-work as a strong economic signal to secure a currency was by vishnumurthy et al. [2003]. in this instance, the token was used to keep peer-to-peer file trading in check, providing “consumers” with the ability to make micro-payments to “suppliers” for their services. the security model afforded by the proof-of-work was augmented with digital signatures and a ledger in order to ensure that the historical record couldn’t be corrupted and that malicious actors could not spoof payment or unjustly complain about service delivery. five years later, nakamoto [2008] introduced another such proof-of-work-secured value token, somewhat wider in scope. the fruits of this project, bitcoin, became the first widely adopted global decentralised transaction ledger. other projects built on bitcoin’s success; the alt-coins introduced numerous other currencies through alteration to the protocol. some of the best known are litecoin and primecoin, discussed by sprankel [2013]. other projects sought to take the core value content mechanism of the protocol and repurpose it; aron [2012] discusses, for example, 1 ethereum: a secure decentralised generalised transaction ledger shanghai version 2 the namecoin project which aims to provide a decentralised name-resolution system. other projects still aim to build upon the bitcoin network itself, leveraging the large amount of value placed in the system and the vast amount of computation that goes into the consensus mechanism. the mastercoin project, first proposed by willett [2013], aims to build a richer protocol involving many additional high-level features on top of the bitcoin protocol through utilisation of a number of auxiliary parts to the core protocol. the coloured coins project, proposed by rosenfeld et al. [2012], takes a similar but more simplified strategy, embellishing the rules of a transaction in order to break the fungibility of bitcoin’s base currency and allow the creation and tracking of tokens through a special “chroma-wallet”-protocol-aware piece of software. additional work has been done in the area with discarding the decentralisation foundation; ripple, discussed by boutellier and heinzen [2014], has sought to create a “federated” system for currency exchange, effectively creating a new financial clearing system. it has demonstrated that high efficiency gains can be made if the decentralisation premise is discarded. early work on smart contracts has been done by szabo [1997] and miller [1997]. around the 1990s it became clear that algorithmic enforcement of agreements could become a significant force in human cooperation. though no specific system was proposed to implement such a system, it was proposed that the future of law would be heavily affected by such systems. in this light, ethereum may be seen as a general implementation of such a crypto-law system. for a list of terms used in this paper, refer to appendix a. 2. the blockchain paradigm ethereum, taken as a whole, can be viewed as a transaction-based state machine: we begin with a genesis state and incrementally execute transactions to morph it into some current state. it is this current state which we accept as the canonical “version” of the world of ethereum. the state can include such information as account balances, reputations, trust arrangements, data pertaining to information of the physical world; in short, anything that can currently be represented by a computer is admissible. transactions thus represent a valid arc between two states; the ‘valid’ part is important—there exist far more invalid state changes than valid state changes. invalid state changes might, e.g., be things such as reducing an account balance without an equal and opposite increase elsewhere. a valid state transition is one which comes about through a transaction. formally: (1) σt+1 ≡ υ(σt,t) where υ is the ethereum state transition function. in ethereum, υ, together with σ are considerably more powerful than any existing comparable system; υ allows components to carry out arbitrary computation, while σ allows components to store arbitrary state between transactions. transactions are collated into blocks; blocks are chained together using a cryptographic hash as a means of reference. blocks function as a journal, recording a series of transactions together with the previous block and an identifier for the final state (though do not store the final state itself—that would be far too big). formally, we expand to: σt+1 ≡ π(σt,b)(2) b ≡ (..., (t0,t1, ...), ...)(3) π(σ,b) ≡ υ(υ(σ,t0),t1)...(4) where b is this block, which includes a series of transactions amongst some other components and π is the block-level state-transition function for transactions1. this is the basis of the blockchain paradigm, a model that forms the backbone of not only ethereum, but all decentralised consensus-based transaction systems to date. 2.1. value. in order to incentivise computation within the network, there needs to be an agreed method for transmitting value. to address this issue, ethereum has an intrinsic currency, ether, known also as eth and sometimes referred to by the old english d̄. the smallest subdenomination of ether, and thus the one in which all integer values of the currency are counted, is the wei. one ether is defined as being 1018 wei. there exist other subdenominations of ether: multiplier name 100 wei 109 gwei 1012 szabo 1015 finney 1018 ether throughout the present work, any reference to value, in the context of ether, currency, a balance or a payment, should be assumed to be counted in wei. 2.2. which history? since the system is decentralised and all parties have an opportunity to create a new block on some older pre-existing block, the resultant structure is necessarily a tree of blocks. in order to form a consensus as to which path, from root (the genesis block) to leaf (the block containing the most recent transactions) through this tree structure, known as the blockchain, there must be an agreed-upon scheme. if there is ever a disagreement between nodes as to which root-to-leaf path down the block tree is the ‘best’ blockchain, then a fork occurs. this would mean that past a given point in time (block), multiple states of the system may coexist: some nodes believing one block to contain the canonical transactions, other nodes believing some other block to be canonical, potentially containing radically different or incompatible transactions. this is to be avoided at all costs as the uncertainty that would ensue would likely kill all confidence in the entire system. since the paris hard fork, reaching consensus on new blocks is managed by a protocol called the beacon chain. it is known as the consensus layer of ethereum, and it defines the rules for determining the canonical history of ethereum blocks. this document describes the execution layer of ethereum. the execution layer defines the rules for interacting with and updating the state of the ethereum virtual machine. the consensus layer is described in greater detail in the consensus specifications. how the consensus 1note that since the shanghai fork, blocks also needs to process withdrawal operations in order to reach their final state. withdrawal operations are defined in sub-section 4.3, and block final state is discussed in greater detail in section 12. https://github.com/ethereum/consensus-specs ethereum: a secure decentralised generalised transaction ledger shanghai version 3 layer is used to determine the canonical state of ethereum is discussed in section 11. there are many versions of ethereum, as the protocol has undergone a number of updates. these updates can be specified to occur: • at a particular block number in the case of preparis updates, • after reaching a terminal total difficulty in the case of the paris update, or • at a particular block timestamp in the case of post-paris updates. this document describes the shanghai version. in order to follow back the history of a path, one must reference multiple versions of this document. here are the block numbers of protocol updates on the ethereum main network:2 name first block number fhomestead 1150000 ftangerinewhistle 2463000 fspuriousdragon 2675000 fbyzantium 4370000 fconstantinople 7280000 fpetersburg 7280000 fistanbul 9069000 fmuirglacier 9200000 fberlin 12244000 flondon 12965000 farrowglacier 13773000 fgrayglacier 15050000 fparis 15537394 fshanghai 17034870 occasionally actors do not agree on a protocol change, and a permanent fork occurs. in order to distinguish between diverged blockchains, eip-155 by buterin [2016] introduced the concept of chain id, which we denote by β. for the ethereum main network (5) β = 1 3. conventions we use a number of typographical conventions for the formal notation, some of which are quite particular to the present work: the two sets of highly structured, ‘top-level’, state values, are denoted with bold lowercase greek letters. they fall into those of world-state, which are denoted σ (or a variant thereupon) and those of machine-state, µ. functions operating on highly structured values are denoted with an upper-case greek letter, e.g. υ, the ethereum state transition function. for most functions, an uppercase letter is used, e.g. c, the general cost function. these may be subscripted to denote specialised variants, e.g. csstore, the cost function for the sstore operation. for specialised and possibly externally defined functions, we may format as typewriter text, e.g. the keccak-256 hash function (as per version 3 of the winning entry to the sha-3 contest by bertoni et al. [2011], rather than the final sha-3 specification), is denoted kec (and generally referred to as plain keccak). also, kec512 refers to the keccak-512 hash function. tuples are typically denoted with an upper-case letter, e.g. t, is used to denote an ethereum transaction. this symbol may, if accordingly defined, be subscripted to refer to an individual component, e.g. tn, denotes the nonce of said transaction. the form of the subscript is used to denote its type; e.g. uppercase subscripts refer to tuples with subscriptable components. scalars and fixed-size byte sequences (or, synonymously, arrays) are denoted with a normal lower-case letter, e.g. n is used in the document to denote a transaction nonce. those with a particularly special meaning may be greek, e.g. δ, the number of items required on the stack for a given operation. arbitrary-length sequences are typically denoted as a bold lower-case letter, e.g. o is used to denote the byte sequence given as the output data of a message call. for particularly important values, a bold uppercase letter may be used. throughout, we assume scalars are non-negative integers and thus belong to the set n. the set of all byte sequences is b, formally defined in appendix b. if such a set of sequences is restricted to those of a particular length, it is denoted with a subscript, thus the set of all byte sequences of length 32 is named b32 and the set of all non-negative integers smaller than 2256 is named n256. this is formally defined in section 4.4. square brackets are used to index into and reference individual components or subsequences of sequences, e.g. µs[0] denotes the first item on the machine’s stack. for subsequences, ellipses are used to specify the intended range, to include elements at both limits, e.g. µm[0..31] denotes the first 32 items of the machine’s memory. in the case of the global state σ, which is a sequence of accounts, themselves tuples, the square brackets are used to reference an individual account. when considering variants of existing values, we follow the rule that within a given scope for definition, if we assume that the unmodified ‘input’ value be denoted by the placeholder � then the modified and utilisable value is denoted as �′, and intermediate values would be �∗, �∗∗ &c. on very particular occasions, in order to maximise readability and only if unambiguous in meaning, we may use alpha-numeric subscripts to denote intermediate values, especially those of particular note. when considering the use of existing functions, given a function f, the function f∗ denotes a similar, element-wise version of the function mapping instead between sequences. it is formally defined in section 4.4. we define a number of useful functions throughout. one of the more common is `, which evaluates to the last item in the given sequence: (6) `(x) ≡ x[‖x‖− 1] 4. blocks, state and transactions having introduced the basic concepts behind ethereum, we will discuss the meaning of a transaction, a block and the state in more detail. 2note that while the paris, shanghai, and every upcoming forks activated at a given block number (e.g. 15,537,394 for paris), the trigger was not the block number, but rather reaching a specified timestamp (or total difficulty for paris). the trigger for the paris and subsequent hard fork are discussed in greater detail in section 10. ethereum: a secure decentralised generalised transaction ledger shanghai version 4 4.1. world state. the world state (state), is a mapping between addresses (160-bit identifiers) and account states (a data structure serialised as rlp, see appendix b). though not stored on the blockchain, it is assumed that the implementation will maintain this mapping in a modified merkle patricia tree (trie, see appendix d). the trie requires a simple database backend that maintains a mapping of byte arrays to byte arrays; we name this underlying database the state database. this has a number of benefits; firstly the root node of this structure is cryptographically dependent on all internal data and as such its hash can be used as a secure identity for the entire system state. secondly, being an immutable data structure, it allows any previous state (whose root hash is known) to be recalled by simply altering the root hash accordingly. since we store all such root hashes in the blockchain, we are able to trivially revert to old states. the account state, σ[a], comprises the following four fields: nonce: a scalar value equal to the number of transactions sent from this address or, in the case of accounts with associated code, the number of contract-creations made by this account. for account of address a in state σ, this would be formally denoted σ[a]n. balance: a scalar value equal to the number of wei owned by this address. formally denoted σ[a]b. storageroot: a 256-bit hash of the root node of a merkle patricia tree that encodes the storage contents of the account (a mapping between 256-bit integer values), encoded into the trie as a mapping from the keccak 256-bit hash of the 256-bit integer keys to the rlp-encoded 256-bit integer values. the hash is formally denoted σ[a]s. codehash: the hash of the evm code of this account—this is the code that gets executed should this address receive a message call. all such code fragments are contained in the state database under their corresponding hashes for later retrieval. this hash is formally denoted σ[a]c, and thus the code may be denoted as b, given that kec(b) = σ[a]c. since we typically wish to refer not to the trie’s root hash but to the underlying set of key/value pairs stored within, we define a convenient equivalence: (7) trie ( l ∗ i (σ[a]s) ) ≡ σ[a]s the collapse function for the set of key/value pairs in the trie, l∗i , is defined as the element-wise transformation of the base function li, given as: (8) li ( (k,v) ) ≡ ( kec(k),rlp(v) ) where: (9) k ∈ b32 ∧ v ∈ n it shall be understood that σ[a]s is not a ‘physical’ member of the account and does not contribute to its later serialisation. if the codehash field is the keccak-256 hash of the empty string, i.e. σ[a]c = kec ( () ) , then the node represents a simple account, sometimes referred to as a “non-contract” account. thus we may define a world-state collapse function ls: (10) ls(σ) ≡{p(a) : σ[a] 6= ∅} where (11) p(a) ≡ ( kec(a),rlp ( (σ[a]n,σ[a]b,σ[a]s,σ[a]c) )) this function, ls, is used alongside the trie function to provide a short identity (hash) of the world state. we assume: (12) ∀a : σ[a] = ∅ ∨ (a ∈ b20 ∧ v(σ[a])) where v is the account validity function: (13) v(x) ≡ xn ∈ n256∧xb ∈ n256∧xs ∈ b32∧xc ∈ b32 an account is empty when it has no code, zero nonce and zero balance: (14) empty(σ,a) ≡ σ[a]c = kec ( () ) ∧σ[a]n = 0∧σ[a]b = 0 even callable precompiled contracts can have an empty account state. this is because their account states do not usually contain the code describing its behavior. an account is dead when its account state is non-existent or empty: (15) dead(σ,a) ≡ σ[a] = ∅∨empty(σ,a) 4.2. the transaction. a transaction (formally, t) is a single cryptographically-signed instruction constructed by an actor externally to the scope of ethereum. the sender of a transaction cannot be a contract. while it is assumed that the ultimate external actor will be human in nature, software tools will be used in its construction and dissemination3. eip-2718 by zoltu [2020] introduced the notion of different transaction types. as of the london version of the protocol, there are three transaction types: 0 (legacy), 1 (eip-2930 by buterin and swende [2020b]), and 2 (eip-1559 by buterin et al. [2019]). further, there are two subtypes of transactions: those which result in message calls and those which result in the creation of new accounts with associated code (known informally as ‘contract creation’). all transaction types specify a number of common fields: type: eip-2718 transaction type; formally tx. nonce: a scalar value equal to the number of transactions sent by the sender; formally tn. gaslimit: a scalar value equal to the maximum amount of gas that should be used in executing this transaction. this is paid up-front, before any computation is done and may not be increased later; formally tg. to: the 160-bit address of the message call’s recipient or, for a contract creation transaction, ∅, used here to denote the only member of b0 ; formally tt. value: a scalar value equal to the number of wei to be transferred to the message call’s recipient or, in the case of contract creation, as an endowment to the newly created account; formally tv. 3notably, such ‘tools’ could ultimately become so causally removed from their human-based initiation—or humans may become so causally-neutral—that there could be a point at which they rightly be considered autonomous agents. e.g. contracts may offer bounties to humans for being sent transactions to initiate their execution. ethereum: a secure decentralised generalised transaction ledger shanghai version 5 r, s: values corresponding to the signature of the transaction and used to determine the sender of the transaction; formally tr and ts. this is expanded in appendix f. eip-2930 (type 1) and eip-1559 (type 2) transactions also have: accesslist: list of access entries to warm up; formally ta. each access list entry e is a tuple of an account address and a list of storage keys: e ≡ (ea,es). chainid: chain id; formally tc. must be equal to the network chain id β. yparity: signature y parity; formally ty. legacy transactions do not have an accesslist (ta = ()), while chainid and yparity for legacy transactions are combined into a single value: w: a scalar value encoding y parity and possibly chain id; formally tw. tw = 27 + ty or tw = 2β + 35 +ty (see eip-155 by buterin [2016]). there are differences in how one’s acceptable gas price is specified in type 2 transactions versus type 0 and type 1 transactions. type 2 transactions take better advantage of the gas market improvements introduced in eip-1559 by explicitly limiting the priority fee4 that is paid. type 2 transactions have the following two fields related to gas: maxfeepergas: a scalar value equal to the maximum number of wei to be paid per unit of gas for all computation costs incurred as a result of the execution of this transaction; formally tm. maxpriorityfeepergas: a scalar value equal to the maximum number of wei to be paid to the block’s fee recipient as an incentive to include the transaction; formally tf . in contrast, type 0 and type 1 transactions specify gas price as a single value: gasprice: a scalar value equal to the number of wei to be paid per unit of gas for all computation costs incurred as a result of the execution of this transaction; formally tp. 5 additionally, a contract creation transaction (regardless of transaction type) contains: init: an unlimited size byte array specifying the evm-code for the account initialisation procedure, formally ti. init is an evm-code fragment; it returns the body, a second fragment of code that executes each time the account receives a message call (either through a transaction or due to the internal execution of code). init is executed only once at account creation and gets discarded immediately thereafter. in contrast, a message call transaction contains: data: an unlimited size byte array specifying the input data of the message call, formally td. appendix f specifies the function, s, which maps transactions to the sender, and happens through the ecdsa of the secp-256k1 curve, using the hash of the transaction (excepting the latter three signature fields) as the datum to sign. for the present we simply assert that the sender of a given transaction t can be represented with s(t). (16) lt(t) ≡   (tn,tp,tg,tt,tv, p,tw,tr,ts) if tx = 0 (tc,tn,tp,tg,tt,tv, p,ta,ty,tr,ts) if tx = 1 (tc,tn,tf,tm,tg,tt,tv, p,ta,ty,tr,ts) if tx = 2 where (17) p ≡ { ti if tt = ∅ td otherwise here, we assume all components are interpreted by the rlp as integer values, with the exception of the access list ta and the arbitrary length byte arrays ti and td. (18) tx ∈{0, 1, 2} ∧ tc = β ∧ tn ∈ n256 ∧ tp ∈ n256 ∧ tg ∈ n256 ∧ tv ∈ n256 ∧ tw ∈ n256 ∧ tr ∈ n256 ∧ ts ∈ n256 ∧ ty ∈ n1 ∧ td ∈ b ∧ ti ∈ b ∧ tm ∈ n256 ∧ tf ∈ n256 where (19) nn = {p : p ∈ n∧p < 2n} the address hash tt is slightly different: it is either a 20-byte address hash or, in the case of being a contractcreation transaction (and thus formally equal to ∅), it is the rlp empty byte sequence and thus the member of b0: (20) tt ∈ { b20 if tt 6= ∅ b0 otherwise 4.3. the withdrawal. a withdrawal (formally, w ) is a tuple of data describing a consensus layer validator’s withdrawal of some amount of its staked ether. a withdrawal is created and validated in the consensus layer of the blockchain and then pushed to the execution layer. a withdrawal is composed of the following fields: globalindex: zero based incrementing withdrawal index that acts as a unique identifier for this withdrawal; formally wg. validatorindex: index of consensus layer’s validator this withdrawal corresponds to; formally wv. recipient: the 20-byte address that will receives ether form this withdrawal; formally wr. amount: a nonzero amount of ether denominated in gwei (109 wei); formally wa. withdrawal serialisation is defined as: (21) lw(w) ≡ (wg,wv,wr,ta) here, we assume all components are interpreted by the rlp as integer values except for wr which is a 20-byte address: (22) wg ∈ n64 ∧ wv ∈ n64 ∧ wr ∈ b20 ∧ wa ∈ n64 4the priority fee is discussed in greater detail in sections 5 and 6. 5type 0 and type 1 transactions will get the same gas price behavior as a type 2 transaction with tm and tf set to the value of tp. 6ommer is a gender-neutral term to mean “sibling of parent”; see https://nonbinary.miraheze.org/wiki/gender_neutral_language_in_ english#aunt/uncle https://nonbinary.miraheze.org/wiki/gender_neutral_language_in_english#aunt/uncle https://nonbinary.miraheze.org/wiki/gender_neutral_language_in_english#aunt/uncle ethereum: a secure decentralised generalised transaction ledger shanghai version 6 4.4. the block. the block in ethereum is the collection of relevant pieces of information (known as the block header ), h, together with information corresponding to the comprised transactions, t, a now deprecated property u that prior to the paris hard fork contained headers of blocks whose parents were equal to the present block’s parent’s parent (such blocks were known as ommers6), and, since the shanghai hard fork, w, a collection of validator’s withdrawal pushed by the consensus layer. the block header contains several pieces of information: parenthash: the keccak 256-bit hash of the parent block’s header, in its entirety; formally hp. ommershash: a 256-bit hash field that is now deprecated due to the replacement of proof of work consensus. it is now to a constant, kec(rlp(())); formally ho. beneficiary: the 160-bit address to which priority fees from this block are transferred; formally hc. stateroot: the keccak 256-bit hash of the root node of the state trie, after all transactions and withdrawals are executed and finalisations applied; formally hr. transactionsroot: the keccak 256-bit hash of the root node of the trie structure populated with each transaction in the transactions list portion of the block; formally ht. receiptsroot: the keccak 256-bit hash of the root node of the trie structure populated with the receipts of each transaction in the transactions list portion of the block; formally he. logsbloom: the bloom filter composed from indexable information (logger address and log topics) contained in each log entry from the receipt of each transaction in the transactions list; formally hb. difficulty: a scalar field that is now deprecated due to the replacement of proof of work consensus. it is set to 0; formally hd. number: a scalar value equal to the number of ancestor blocks. the genesis block has a number of zero; formally hi. gaslimit: a scalar value equal to the current limit of gas expenditure per block; formally hl. gasused: a scalar value equal to the total gas used in transactions in this block; formally hg. timestamp: a scalar value equal to the reasonable output of unix’s time() at this block’s inception; formally hs. extradata: an arbitrary byte array containing data relevant to this block. this must be 32 bytes or fewer; formally hx. prevrandao: the latest randao mix7 of the post beacon state of the previous block; formally ha. nonce: a 64-bit value that is now deprecated due to the replacement of proof of work consensus. it is set to 0x0000000000000000; formally hn. basefeepergas: a scalar value equal to the amount of wei that is burned for each unit of gas consumed; formally hf . withdrawalsroot: the keccak 256-bit hash of the root node of the trie structure populated with each withdrawal operations pushed by the consensus layer for this block; formally hw. the other three components in the block are a series of transactions, bt, an empty array which was previously reserved for ommer block headers, bu, and a series of withdrawals, bw. formally, we can refer to a block b: (23) b ≡ (bh,bt,bu,bw) 4.4.1. transaction receipt. in order to encode information about a transaction concerning which it may be useful to form a zero-knowledge proof, or index and search, we encode a receipt of each transaction containing certain information from its execution. each receipt, denoted br[i] for the ith transaction, is placed in an index-keyed trie and the root recorded in the header as he. the transaction receipt, r, is a tuple of five items comprising: the type of the transaction, rx, the status code of the transaction, rz, the cumulative gas used in the block containing the transaction receipt as of immediately after the transaction has happened, ru, the set of logs created through execution of the transaction, rl and the bloom filter composed from information in those logs, rb: (24) r ≡ (rx,rz,ru,rb,rl) rx is equal to the type of the corresponding transaction. the function lr prepares a transaction receipt for being transformed into an rlp-serialised byte array: (25) lr(r) ≡ (rz,ru,rb,rl) we assert that the status code rz is a non-negative integer: (26) rz ∈ n we assert that ru, the cumulative gas used, is a nonnegative integer and that the logs bloom, rb, is a hash of size 2048 bits (256 bytes): (27) ru ∈ n ∧ rb ∈ b256 the sequence rl is a series of log entries, (o0,o1, ...). a log entry, o, is a tuple of the logger’s address, oa, a possibly empty series of 32-byte log topics, ot and some number of bytes of data, od: (28) o ≡ (oa, (ot0,ot1, ...),od) (29) oa ∈ b20 ∧ ∀x ∈ ot : x ∈ b32 ∧ od ∈ b we define the bloom filter function, m, to reduce a log entry into a single 256-byte hash: (30) m(o) ≡ ∨ x∈{oa}∪ot ( m3:2048(x) ) where m3:2048 is a specialised bloom filter that sets three bits out of 2048, given an arbitrary byte sequence. it does this through taking the low-order 11 bits of each of the first three pairs of bytes in a keccak-256 hash of the byte 7randao is a pseudorandom value generated by validators on the ethereum consensus layer. refer to the consensus layer specs (https://github.com/ethereum/consensus-specs) for more detail on randao. 82048 = 211(11 bits), and the low-order 11 bits is the modulo 2048 of the operand, which is in this case is “each of the first three pairs of bytes in a keccak-256 hash of the byte sequence.” https://github.com/ethereum/consensus-specs ethereum: a secure decentralised generalised transaction ledger shanghai version 7 sequence.8 formally: m3:2048(x : x ∈ b) ≡ y : y ∈ b256 where:(31) y = (0, 0, ..., 0) except:(32) ∀i ∈{0, 2, 4} : b2047−m(x,i)(y) = 1(33) m(x, i) ≡ kec(x)[i, i + 1] mod 2048(34) where b is the bit reference function such that bj(x) equals the bit of index j (indexed from 0) in the byte array x. notably, it treats x as big-endian (more significant bits will have smaller indices). 4.4.2. holistic validity. we can assert a block’s validity if and only if it satisfies several conditions: the block’s ommers field bu must be an empty array and the block’s header must be consistent with the given transactions bt and withdrawals bw. for the header to be consistent with the transactions bt and withdrawals bw, stateroot (hr) must match the resultant state after executing all transactions, then all withdrawals, in order on the base state σ (as specified in section 12), and transactionsroot (ht), receiptsroot (he), logsbloom (hb) and withdrawalsroot (hw) must be correctly derived from the transactions themselves, the transaction receipts resulting from execution, the resulting logs, and the withdrawals, respectively. (35) bu ≡ () ∧ hr ≡ trie(ls(π(σ,b))) ∧ ht ≡ trie({∀i < ‖bt‖, i ∈ n : pt(i,bt[i])}) ∧ he ≡ trie({∀i < ‖br‖, i ∈ n : pr(i,br[i])}) ∧ hw ≡ trie({∀i < ‖bw‖, i ∈ n : pw(i,bw[i])}) ∧ hb ≡ ∨ r∈br ( rb ) where pw(k,v) is a pairwise rlp transformation for withdrawals: (36) pw(k,w) ≡ (rlp(k),rlp(lw(w))) similarly, pt(k,v) and pr(k,v) are pairwise rlp transformations, but with a special treatment for eip-2718 transactions: (37) pt(k,t) ≡ ( rlp(k), { rlp(lt(t)) if tx = 0 (tx) ·rlp(lt(t)) otherwise ) and (38) pr(k,r) ≡ ( rlp(k), { rlp(lr(r)) if rx = 0 (rx) ·rlp(lr(r)) otherwise ) (· is the concatenation of byte arrays). furthermore: (39) trie(ls(σ)) = p(bh)hr thus trie(ls(σ)) is the root node hash of the merkle patricia tree structure containing the key-value pairs of the state σ with values encoded using rlp, and p (bh) is the parent block of b, defined directly. the values stemming from the computation of transactions, specifically the transaction receipts, br, and that defined through the transaction’s state-accumulation function, π, are formalised later in section 12.3. 4.4.3. serialisation. the function lb and lh are the preparation functions for a block and block header respectively. we assert the types and order of the structure for when the rlp transformation is required: lh(h) ≡ ( hp,ho,hc,hr,ht,he,hb,hd, hi,hl,hg,hs,hx,ha,hn,hf,hw ) (40) lb(b) ≡ ( lh(bh), l̃ ∗ t(bt),l ∗ h(bu),l ∗ w(bw) ) (41) where l̃t takes a special care of eip-2718 transactions: (42) l̃t(t) = { lt(t) if tx = 0 (tx) ·rlp(lt(t)) otherwise with l̃∗t, l ∗ h, and l ∗ w, being element-wise sequence transformations, thus: (43) f ∗( (x0,x1, ...) ) ≡ ( f(x0),f(x1), ... ) for any function f the component types are defined thus: (44) hp ∈ b32 ∧ ho ∈ b32 ∧ hc ∈ b20 ∧ hr ∈ b32 ∧ ht ∈ b32 ∧ he ∈ b32 ∧ hb ∈ b256 ∧ hd ∈ n ∧ hi ∈ n ∧ hl ∈ n ∧ hg ∈ n ∧ hs ∈ n256 ∧ hx ∈ b ∧ ha ∈ b32 ∧ hn ∈ b8 ∧ hf ∈ n ∧ hw ∈ b32 where (45) bn = {b : b ∈ b∧‖b‖ = n} we now have a rigorous specification for the construction of a formal block structure. the rlp function rlp (see appendix b) provides the canonical method for transforming this structure into a sequence of bytes ready for transmission over the wire or storage locally. 4.4.4. block header validity. we define p(bh) to be the parent block of b, formally: (46) p(h) ≡ b′ : kec(rlp(b′h)) = hp the block number is the parent’s block number incremented by one: (47) hi ≡ p(h)hi + 1 the london release introduced the block attribute base fee per gas hf (see eip-1559 by buterin et al. [2019]). the base fee is the amount of wei burned per unit of gas consumed while executing transactions within the block. the value of the base fee is a function of the difference between the gas used by the parent block and the parent block’s gas target. the expected base fee per gas is defined as f(h): (48) f(h) ≡   1000000000 if hi = flondon p(h)hf if p(h)hg = τ p(h)hf −ν if p(h)hg < τ p(h)hf + ν if p(h)hg > τ ethereum: a secure decentralised generalised transaction ledger shanghai version 8 where: τ ≡b p(h)hl ρ c(49) ρ ≡ 2(50) ν ∗ ≡ { b p(h)hf×(τ−p(h)hg) τ c if p(h)hg < τ b p(h)hf×(p(h)hg−τ) τ c if p(h)hg > τ (51) ν ≡ { bν ∗ ξ c if p(h)hg < τ max(bν ∗ ξ c, 1) if p(h)hg > τ (52) ξ ≡ 8(53) the gas target, τ, is defined as the gas limit hl divided by the elasticity multiplier, ρ, a global constant set to 2. so while blocks can consume as much gas as the gas limit, the base fee is adjusted so that on average, blocks consume as much gas as the gas target. the base fee is increased in the current block when the parent block’s gas usage exceeds the gas target, and, conversely, the base fee is decreased in the current block when the parent block’s gas usage is less than the gas target. the magnitude of the increase or decrease in the base fee, defined as ν, is proportional to the difference between the amount of gas the parent block consumed and the parent block’s gas target. the effect on the base fee is dampened by a global constant called the base fee max change denominator, formally ξ, set to 8. a value of 8 entails that the base fee can increase or decrease by at most 12.5% from one block to the next. the canonical gas limit hl of a block of header h must fulfil the relation: hl < p(h)hl′ + ⌊ p(h)hl′ 1024 ⌋ ∧(54) hl > p(h)hl′ − ⌊ p(h)hl′ 1024 ⌋ ∧ hl > 5000 where: p(h)hl′ ≡ { p(h)hl ×ρ if hi = flondon p(h)hl if hi > flondon (55) to avoid a discontinuity in gas usage, the value of the parent block gas limit for the purpose of validating the current block’s gas limit is modified at the london fork block by multiplying it by the elasticity multiplier, ρ. we call this modified value p(h)hl′. this ensures that the gas target for post-london blocks can be set roughly in line with the gas limit of pre-london blocks. hs is the timestamp (in unix’s time()) of block h and must fulfil the relation: (56) hs > p(h)hs the paris hard fork changed ethereum’s consensus from proof of work to proof of stake, and thus deprecated many block header properties related to proof of work. these deprecated properties include nonce (hn), ommershash (ho), difficulty (hd), and mixhash (hm). mixhash has been replaced with a new field prevrandao (ha). the other header fields related to proof of work have been replaced with constants: ho ≡ kec(rlp(()))(57) hd ≡ 0(58) hn ≡ 0x0000000000000000(59) the value of prevrandao must be determined using information from the beacon chain. while the details of generating the randao value on the beacon chain is beyond the scope of this paper, we refer to the expected randao value for the previous block as prevrandao(). thus we are able to define the block header validity function v (h): v (h) ≡ hg ≤ hl ∧(60) hl < p(h)hl′ + ⌊ p(h)hl′ 1024 ⌋ ∧ hl > p(h)hl′ − ⌊ p(h)hl′ 1024 ⌋ ∧ hl > 5000 ∧ hs > p(h)hs ∧ hi = p(h)hi + 1 ∧ ‖hx‖≤ 32 ∧ hf = f(h) ∧ ho = kec(rlp(())) ∧ hd = 0 ∧ hn = 0x0000000000000000 ∧ ha = prevrandao() note additionally that extradata must be at most 32 bytes. 5. gas and payment in order to avoid issues of network abuse and to sidestep the inevitable questions stemming from turing completeness, all programmable computation in ethereum is subject to fees. the fee schedule is specified in units of gas (see appendix g for the fees associated with various computation). thus any given fragment of programmable computation (this includes creating contracts, making message calls, utilising and accessing account storage and executing operations on the virtual machine) has a universally agreed cost in terms of gas. every transaction has a specific amount of gas associated with it: gaslimit. this is the amount of gas which is implicitly purchased from the sender’s account balance. the purchase happens at the effective gas price defined in section 6. the transaction is considered invalid if the account balance cannot support such a purchase. it is named gaslimit since any unused gas at the end of the transaction is refunded (at the same rate of purchase) to the sender’s account. gas does not exist outside of the execution of a transaction. thus for accounts with trusted code associated, a relatively high gas limit may be set and left alone. since the introduction of eip-1559 by buterin et al. [2019] in the london hard fork, every transaction included in a block must pay a base fee, which is specified as wei per unit of gas consumed and is constant for each transaction within a block. the ether that is paid to meet the base fee is burned (taken out of circulation). the base fee adjusts dynamically as a function of the previous ethereum: a secure decentralised generalised transaction ledger shanghai version 9 block’s gas consumption relative to its gas target (a value that is currently half the block’s gas limit, which can be adjusted by validators). if the previous block’s total gas consumption exceeds the gas target, this indicates excess demand for block space at the current base fee, and the base fee is increased. conversely, if the gas consumed in the previous block is lower than the gas target, demand for block space is lower than the gas target at the current base fee, and thus the base fee is decreased. this process of adjusting the base fee should bring the average block’s gas consumption in line with the gas target. refer to section 4.4 for greater detail on how the base fee is set. to incentivize validators to include transactions, there is an additional fee known as a priority fee, which is also specified as wei per unit of gas consumed. the total fee paid by the transactor therefore is the sum of the base fee per gas and the priority fee per gas multiplied by the total gas consumed. ether used to satisfy the priority fee is delivered to the beneficiary address, the address of an account typically under the control of the validator. transactors using type 2 transactions can specify the maximum priority fee they are willing to pay (maxpriorityfeepergas) as well as the max total fee they are willing to pay (maxfeepergas), inclusive of both the priority fee and the base fee. maxfeepergas must be at least as high as the base fee for the transaction to be included in a block. type 0 and type 1 transactions have only one field for specifiying a gas price–gasprice– which also must be at least as high as the base fee for inclusion in a block. the amount by which gasprice is higher than the base fee constitutes the priority fee in the case of a type 0 or type 1 transaction. transactors are free to select any priority fee that they wish, however validators are free to ignore transactions as they choose. a higher priority fee on a transaction will therefore cost the sender more in terms of ether and deliver a greater value to the validator and thus will more likely be selected for inclusion. since there will be a (weighted) distribution of minimum acceptable priority fees, transactors will necessarily have a trade-off to make between lowering the priority fee and maximising the chance that their transaction will be included in a block in a timely manner. 6. transaction execution the execution of a transaction is the most complex part of the ethereum protocol: it defines the state transition function υ. it is assumed that any transactions executed first pass the initial tests of intrinsic validity. these include: (1) the transaction is well-formed rlp, with no additional trailing bytes; (2) the transaction signature is valid; (3) the transaction nonce is valid (equivalent to the sender account’s current nonce); (4) the sender account has no contract code deployed (see eip-3607 by feist et al. [2021]); (5) the gas limit is no smaller than the intrinsic gas, g0, used by the transaction; (6) the sender account balance contains at least the cost, v0, required in up-front payment; (7) the maxfeepergas, tm, in the case of type 2 transactions, or gasprice, tp, in the case of type 0 and type 1 transactions, is greater than or equal to the block’s base fee, hf ; and (8) for type 2 transactions, maxpriorityfeepergas, tf , must be no larger than maxfeepergas, tm. formally, we consider the function υ, with t being a transaction and σ the state: (61) σ ′ = υ(σ,t) thus σ′ is the post-transactional state. we also define υg to evaluate to the amount of gas used in the execution of a transaction, υl to evaluate to the transaction’s accrued log items and υz to evaluate to the status code resulting from the transaction. these will be formally defined later. 6.1. substate. throughout transaction execution, we accrue certain information that is acted upon immediately following the transaction. we call this the accrued transaction substate, or accrued substate for short, and represent it as a, which is a tuple: (62) a ≡ (as,al,at,ar,aa,ak) the tuple contents include as, the self-destruct set: a set of accounts that will be discarded following the transaction’s completion. al is the log series: this is a series of archived and indexable ‘checkpoints’ in vm code execution that allow for contract-calls to be easily tracked by onlookers external to the ethereum world (such as decentralised application front-ends). at is the set of touched accounts, of which the empty ones are deleted at the end of a transaction. ar is the refund balance, increased through using the sstore instruction in order to reset contract storage to zero from some non-zero value. though not immediately refunded, it is allowed to partially offset the total execution costs. finally, eip-2929 by buterin and swende [2020a] introduced aa, the set of accessed account addresses, and ak, the set of accessed storage keys (more accurately, each element of ak is a tuple of a 20-byte account address and a 32-byte storage slot). we define the empty accrued substate a0 to have no self-destructs, no logs, no touched accounts, zero refund balance, all precompiled contracts in the accessed addresses, and no accessed storage: (63) a 0 ≡ (∅, (),∅, 0,π,∅) where π is the set of all precompiled addresses. 6.2. execution. we define intrinsic gas g0, the amount of gas this transaction requires to be paid prior to execution, as follows: g0 ≡ ∑ i∈ti,td { gtxdatazero if i = 0 gtxdatanonzero otherwise (64) + { gtxcreate + r(‖ti‖) if tt = ∅ 0 otherwise + gtransaction + ‖ta‖−1∑ j=0 ( gaccesslistaddress + ‖ta[j]s‖gaccessliststorage ) where ti,td means the series of bytes of the transaction’s associated data and initialisation evm-code, depending on ethereum: a secure decentralised generalised transaction ledger shanghai version 10 whether the transaction is for contract-creation or messagecall. gtxcreate is added if the transaction is contractcreating, but not if it is a message call. gaccesslistaddress and gaccessliststorage are the costs of warming up account and storage access, respectively. g is fully defined in appendix g. we define the initcode cost function, formally r, as the amount of gas that needs to be paid for each word of the initcode prior to executing the creation code of a new contract: (65) r(x) ≡ ginitcodeword ×d x 32 e we define the effective gas price, formally p, as the amount of wei the transaction signer will pay per unit of gas consumed during the transaction’s execution. it is calculated as follows: (66) p ≡ { tp if tx = 0 ∨tx = 1 f + hf if tx = 2 where f is the priority fee—the amount of wei the block’s beneficiary address will receive per unit of gas consumed during the transaction’s execution. it is calculated as: (67) f ≡ { tp −hf if tx = 0 ∨tx = 1 min(tf,tm −hf ) if tx = 2 the up-front cost v0 is calculated as: (68) v0 ≡ { tgtp + tv if tx = 0 ∨tx = 1 tgtm + tv if tx = 2 the validity is determined as: (69) s(t) 6= ∅ ∧ σ[s(t)]c = kec ( () ) ∧ tn = σ[s(t)]n ∧ g0 6 tg ∧ v0 6 σ[s(t)]b ∧ m > hf ∧ n 6 49152 ∧ tg 6 bhl − `(br)u where (70) m ≡ { tp if tx = 0 ∨tx = 1 tm if tx = 2 and (71) n ≡ { ‖ti‖ if tt 6= ∅ 0 otherwise the penultimate condition ensures that, for create transactions, the length of the initcode is no greater than 49152 bytes. note the final condition; the sum of the transaction’s gas limit, tg, and the gas utilised in this block prior, given by `(br)u, must be no greater than the block’s gaslimit, bhl. also, with a slight abuse of notation, we assume that σ[s(t)]c = kec ( () ) , σ[s(t)]n = 0, and σ[s(t)]b = 0 if σ[s(t)] = ∅. for type 2 transactions, we add an additional check that maxpriorityfeepergas is no larger than maxfeepergas: (72) tm > tf the execution of a valid transaction begins with an irrevocable change made to the state: the nonce of the account of the sender, s(t ), is incremented by one and the balance is reduced by part of the up-front cost, tgp. the gas available for the proceeding computation, g, is defined as tg −g0. the computation, whether contract creation or a message call, results in an eventual state (which may legally be equivalent to the current state), the change to which is deterministic and never invalid: there can be no invalid transactions from this point. we define the checkpoint state σ0: σ0 ≡ σ except:(73) σ0[s(t)]b ≡ σ[s(t)]b −tgp(74) σ0[s(t)]n ≡ σ[s(t)]n + 1(75) evaluating σp from σ0 depends on the transaction type; either contract creation or message call; we define the tuple of post-execution provisional state σp, remaining gas g ′, accrued substate a and status code z: (76) (σp,g ′ ,a,z) ≡   λ4(σ0,a ∗,s(t),s(t),g, p,tv,ti, 0,∅,>) if tt = ∅ θ4(σ0,a ∗,s(t),s(t),tt, tt,g,p,tv,tv,td, 0,>) otherwise where a ∗ ≡ a0 except(77) a ∗ k ≡ ⋃ e∈ta { ∀i < ‖es‖, i ∈ n : (ea,es[i]) } (78) a ∗ a ≡ { a∪tt if tt 6= ∅ a otherwise (79) a ≡ a0a ∪s(t) ∪hc ∪e∈ta {ea}(80) and g is the amount of gas remaining after deducting the basic amount required to pay for the existence of the transaction: (81) g ≡ tg −g0 note we use θ4 and λ4 to denote the fact that only the first four components of the functions’ values are taken; the final represents the message-call’s output value (a byte array) and is unused in the context of transaction evaluation. then the state is finalised by determining the amount to be refunded, g∗ from the remaining gas, g′, plus some allowance from the refund counter, to the sender at the original rate. (82) g ∗ ≡ g′ + min {⌊ tg −g′ 5 ⌋ ,ar } the total refundable amount is the legitimately remaining gas g′, added to ar, with the latter component being capped up to a maximum of one fifth9 (rounded down) of the total amount used tg −g′. therefore, g∗ is the total gas that remains after the transaction has been executed. 9the max refundable proportion of gas was reduced from one half to one fifth by eip-3529 by buterin and swende [2021] in the london release ethereum: a secure decentralised generalised transaction ledger shanghai version 11 the validator, whose address is specified as the beneficiary of the present block b, receives the gas consumed multiplied by the transaction’s priority fee per gas, defined as f in this section. the ether that is paid by the transactor that goes toward the base fee is debited from the transactor’s account but credited to no other account, so it is burned. we define the pre-final state σ∗ in terms of the provisional state σp: σ ∗ ≡ σp except(83) σ ∗ [s(t)]b ≡ σp[s(t)]b + g∗p(84) σ ∗ [bhc]b ≡ σp[bhc]b + (tg −g ∗ )f(85) the final state, σ′, is reached after deleting all accounts that either appear in the self-destruct set or are touched and empty: σ ′ ≡ σ∗ except(86) ∀i ∈ as : σ′[i] = ∅(87) ∀i ∈ at : σ′[i] = ∅ if dead(σ∗, i)(88) and finally, we specify υg, the total gas used in this transaction υl, the logs created by this transaction and υz, the status code of this transaction: υ g (σ,t) ≡ tg −g∗(89) υ l (σ,t) ≡ al(90) υ z (σ,t) ≡ z(91) these are used to help define the transaction receipt and are also used later for state validation. 7. contract creation there are a number of intrinsic parameters used when creating an account: sender (s), original transactor10 (o), available gas (g), effective gas price (p), endowment (v) together with an arbitrary length byte array, i, the initialisation evm code, the present depth of the messagecall/contract-creation stack (e), the salt for new account’s address (ζ) and finally the permission to make modifications to the state (w). the salt ζ might be missing (ζ = ∅); formally, (92) ζ ∈ b32 ∪b0 if the creation was caused by create2, then ζ 6= ∅. we define the creation function formally as the function λ, which evaluates from these values, together with the state σ and the accrued substate a, to the tuple containing the new state, remaining gas, new accrued substate, status code and output (σ′,g′,a′,z, o): (93) (σ ′ ,g ′ ,a ′ ,z, o) ≡ λ(σ,a,s,o,g,p,v, i,e,ζ,w) the address of the new account is defined as being the rightmost 160 bits of the keccak-256 hash of the rlp encoding of the structure containing only the sender and the account nonce. for create2 the rule is different and is described in eip-1014 by buterin [2018]. combining the two cases, we define the resultant address for the new account a: a ≡ addr(s,σ[s]n − 1,ζ, i)(94) addr(s,n,ζ, i) ≡b96..255 ( kec ( la(s,n,ζ, i) )) (95) la(s,n,ζ, i) ≡ { rlp ( (s,n) ) if ζ = ∅ (255) ·s · ζ ·kec(i) otherwise (96) where · is the concatenation of byte arrays, ba..b(x) evaluates to a binary value containing the bits of indices in the range [a,b] of the binary data x, and σ[x] is the address state of x, or ∅ if none exists. note we use one fewer than the sender’s nonce value; we assert that we have incremented the sender account’s nonce prior to this call, and so the value used is the sender’s nonce at the beginning of the responsible transaction or vm operation. the address of the new account is added to the set of accessed accounts: (97) a ∗ ≡ a except a∗a ≡ aa ∪{a} the account’s nonce is initially defined as one, the balance as the value passed, the storage as empty and the code hash as the keccak 256-bit hash of the empty string; the sender’s balance is also reduced by the value passed. thus the mutated state becomes σ∗: (98) σ ∗ ≡ σ except: σ ∗ [a] = ( 1,v + v ′ ,trie(∅),kec ( () )) (99) σ ∗ [s] = { ∅ if σ[s] = ∅ ∧ v = 0 a∗ otherwise (100) a ∗ ≡ (σ[s]n,σ[s]b −v,σ[s]s,σ[s]c)(101) where v′ is the account’s pre-existing value, in the event it was previously in existence: (102) v ′ ≡ { 0 if σ[a] = ∅ σ[a]b otherwise finally, the account is initialised through the execution of the initialising evm code i according to the execution model (see section 9). code execution can effect several events that are not internal to the execution state: the account’s storage can be altered, further accounts can be created and further message calls can be made. as such, the code execution function ξ evaluates to a tuple of the resultant state σ∗∗, available gas remaining g∗∗, the resultant accrued substate a∗∗ and the body code of the account o. (103) (σ ∗∗ ,g ∗∗ ,a ∗∗ , o) ≡ ξ(σ∗,g,a∗,i) where i contains the parameters of the execution environment, that is: 10which can differ from the sender in the case of a message call or contract creation not directly triggered by a transaction but coming from the execution of evm-code ethereum: a secure decentralised generalised transaction ledger shanghai version 12 ia ≡ a(104) io ≡ o(105) ip ≡ p(106) id ≡ ()(107) is ≡ s(108) iv ≡ v(109) ib ≡ i(110) ie ≡ e(111) iw ≡ w(112) id evaluates to the empty tuple as there is no input data to this call. ih has no special treatment and is determined from the blockchain. code execution depletes gas, and gas may not go below zero, thus execution may exit before the code has come to a natural halting state. in this (and several other) exceptional cases we say an out-of-gas (oog) exception has occurred: the evaluated state is defined as being the empty set, ∅, and the entire create operation should have no effect on the state, effectively leaving it as it was immediately prior to attempting the creation. if the initialization code completes successfully, a final contract-creation cost is paid, the code-deposit cost, c, proportional to the size of the created contract’s code: (113) c ≡ gcodedeposit ×‖o‖ if there is not enough gas remaining to pay this, i.e. g∗∗ < c, then we also declare an out-of-gas exception. the gas remaining will be zero in any such exceptional condition, i.e. if the creation was conducted as the reception of a transaction, then this doesn’t affect payment of the intrinsic cost of contract creation; it is paid regardless. however, the value of the transaction is not transferred to the aborted contract’s address when we are out-of-gas, thus the contract’s code is not stored. if such an exception does not occur, then the remaining gas is refunded to the originator and the now-altered state is allowed to persist. thus formally, we may specify the resultant state, gas, accrued substate and status code as (σ′,g′,a′,z) where: g ′ ≡ { 0 if f g∗∗ − c otherwise (114) σ ′ ≡   σ if f ∨ σ∗∗ = ∅ σ∗∗ except: σ′[a] = ∅ if dead(σ∗∗,a) σ∗∗ except: σ′[a]c = kec(o) otherwise (115) a ′ ≡ { a∗ if f ∨ σ∗∗ = ∅ a∗∗ otherwise (116) z ≡ { 0 if f ∨ σ∗∗ = ∅ 1 otherwise (117) where f ≡ ( σ[a] 6= ∅ ∧ ( σ[a]c 6= kec ( () ) ∨σ[a]n 6= 0 )) ∨ (118) (σ ∗∗ = ∅ ∧ o = ∅) ∨ g ∗∗ < c ∨ ‖o‖ > 24576 ∨ o[0] = 0xef note the last condition of f indicates that contract code cannot begin with the byte 0xef (refer to eip-3541 by beregszaszi et al. [2021]), the exception in the determination of σ′ dictates that o, the resultant byte sequence from the execution of the initialisation code, specifies the final body code for the newly-created account. note that intention is that the result is either a successfully created new contract with its endowment, or no new contract with no transfer of value. in addition, observe that if the execution of the initialising code reverts (σ∗∗ = ∅ ∧ o 6= ∅), the resultant gas g′ is not depleted (provided there was no other exception), but no new account is created. 7.1. subtleties. note that while the initialisation code is executing, the newly created address exists but with no intrinsic body code11. thus any message call received by it during this time causes no code to be executed. if the initialisation execution ends with a selfdestruct instruction, the matter is moot since the account will be deleted before the transaction is completed. for a normal stop code, or if the code returned is otherwise empty, then the state is left with a zombie account, and any remaining balance will be locked into the account forever. 8. message call in the case of executing a message call, several parameters are required: sender (s), transaction originator (o), recipient (r), the account whose code is to be executed 11during initialization code execution, extcodesize on the address should return zero, which is the length of the code of the account while codesize should return the length of the initialization code (as defined in h.2). ethereum: a secure decentralised generalised transaction ledger shanghai version 13 (c, usually the same as recipient), available gas (g), value (v) and effective gas price (p) together with an arbitrary length byte array, d, the input data of the call, the present depth of the message-call/contract-creation stack (e) and finally the permission to make modifications to the state (w). aside from evaluating to a new state and accrued transaction substate, message calls also have an extra component—the output data denoted by the byte array o. this is ignored when executing transactions, however message calls can be initiated due to vm-code execution and in this case this information is used. (119) (σ ′ ,g ′ ,a ′ ,z, o) ≡ θ(σ,a,s,o,r,c,g,p,v, ṽ, d,e,w) note that we need to differentiate between the value that is to be transferred, v, from the value apparent in the execution context, ṽ, for the delegatecall instruction. we define σ1, the first transitional state as the original state but with the value transferred from sender to recipient: (120) σ1[r]b ≡ σ[r]b + v ∧ σ1[s]b ≡ σ[s]b −v unless s = r. throughout the present work, it is assumed that if σ1[r] was originally undefined, it will be created as an account with no code or state and zero balance and nonce. thus the previous equation should be taken to mean: (121) σ1 ≡ σ′1 except: (122) σ1[s] ≡ { ∅ if σ′1[s] = ∅ ∧ v = 0 a1 otherwise (123) a1 ≡ ( σ ′ 1[s]n,σ ′ 1[s]b −v,σ ′ 1[s]s,σ ′ 1[s]c ) (124) and σ ′ 1 ≡ σ except: (125)  σ′1[r] ≡ (0,v,trie(∅),kec(())) if σ[r] = ∅∧v 6= 0 σ′1[r] ≡ ∅ if σ[r] = ∅∧v = 0 σ′1[r] ≡ a′1 otherwise (126) a ′ 1 ≡ (σ[r]n,σ[r]b + v,σ[r]s,σ[r]c) the account’s associated code (identified as the fragment whose keccak-256 hash is σ[c]c) is executed according to the execution model (see section 9). just as with contract creation, if the execution halts in an exceptional fashion (i.e. due to an exhausted gas supply, stack underflow, invalid jump destination or invalid instruction), then no gas is refunded to the caller and the state is reverted to the point immediately prior to balance transfer (i.e. σ). σ ′ ≡ { σ if σ∗∗ = ∅ σ∗∗ otherwise (127) g ′ ≡   0 if σ∗∗ = ∅ ∧ o = ∅ g∗∗ otherwise (128) a ′ ≡ { a if σ∗∗ = ∅ a∗∗ otherwise (129) z ≡ { 0 if σ∗∗ = ∅ 1 otherwise (130) (σ ∗∗ ,g ∗∗ ,a ∗∗ , o) ≡ ξ(131) ia ≡ r(132) io ≡ o(133) ip ≡ p(134) id ≡ d(135) is ≡ s(136) iv ≡ ṽ(137) ie ≡ e(138) iw ≡ w(139) where (140) ξ ≡   ξecrec(σ1,g,a,i) if c = 1 ξsha256(σ1,g,a,i) if c = 2 ξrip160(σ1,g,a,i) if c = 3 ξid(σ1,g,a,i) if c = 4 ξexpmod(σ1,g,a,i) if c = 5 ξbn add(σ1,g,a,i) if c = 6 ξbn mul(σ1,g,a,i) if c = 7 ξsnarkv(σ1,g,a,i) if c = 8 ξblake2 f(σ1,g,a,i) if c = 9 ξ(σ1,g,a,i) otherwise and (141) kec(ib) = σ[c]c it is assumed that the client will have stored the pair (kec(ib),ib) at some point prior in order to make the determination of ib feasible. as can be seen, there are nine exceptions to the usage of the general execution framework ξ for evaluation of the message call: these are so-called ‘precompiled’ contracts, meant as a preliminary piece of architecture that may later become native extensions. the contracts in addresses 1 to 9 execute the elliptic curve public key recovery function, the sha2 256-bit hash scheme, the ripemd 160-bit hash scheme, the identity function, arbitrary precision modular exponentiation, elliptic curve addition, elliptic curve scalar multiplication, an elliptic curve pairing check, and the blake2 compression function f respectively. their full formal definition is in appendix e. we denote the set of the addresses of the precompiled contracts by π: (142) π ≡{1, 2, 3, 4, 5, 6, 7, 8, 9} 9. execution model the execution model specifies how the system state is altered given a series of bytecode instructions and a small tuple of environmental data. this is specified through a ethereum: a secure decentralised generalised transaction ledger shanghai version 14 formal model of a virtual state machine, known as the ethereum virtual machine (evm). it is a quasi-turingcomplete machine; the quasi qualification comes from the fact that the computation is intrinsically bounded through a parameter, gas, which limits the total amount of computation done. 9.1. basics. the evm is a simple stack-based architecture. the word size of the machine (and thus size of stack items) is 256-bit. this was chosen to facilitate the keccak256 hash scheme and elliptic-curve computations. the memory model is a simple word-addressed byte array. the stack has a maximum size of 1024. the machine also has an independent storage model; this is similar in concept to the memory but rather than a byte array, it is a wordaddressable word array. unlike memory, which is volatile, storage is non volatile and is maintained as part of the system state. all locations in both storage and memory are well-defined initially as zero. the machine does not follow the standard von neumann architecture. rather than storing program code in generally-accessible memory or storage, it is stored separately in a virtual rom interactable only through a specialised instruction. the machine can have exceptional execution for several reasons, including stack underflows and invalid instructions. like the out-of-gas exception, they do not leave state changes intact. rather, the machine halts immediately and reports the issue to the execution agent (either the transaction processor or, recursively, the spawning execution environment) which will deal with it separately. 9.2. fees overview. fees (denominated in gas) are charged under three distinct circumstances, all three as prerequisite to the execution of an operation. the first and most common is the fee intrinsic to the computation of the operation (see appendix g). secondly, gas may be deducted in order to form the payment for a subordinate message call or contract creation; this forms part of the payment for create, create2, call and callcode. finally, gas may be paid due to an increase in the usage of the memory. over an account’s execution, the total fee for memoryusage payable is proportional to smallest multiple of 32 bytes that are required such that all memory indices (whether for read or write) are included in the range. this is paid for on a just-in-time basis; as such, referencing an area of memory at least 32 bytes greater than any previously indexed memory will certainly result in an additional memory usage fee. due to this fee it is highly unlikely addresses will ever go above 32-bit bounds. that said, implementations must be able to manage this eventuality. storage fees have a slightly nuanced behaviour—to incentivise minimisation of the use of storage (which corresponds directly to a larger state database on all nodes), the execution fee for an operation that clears an entry in the storage is not only waived, a qualified refund is given; in fact, this refund is effectively paid up-front since the initial usage of a storage location costs substantially more than normal usage. see appendix h for a rigorous definition of the evm gas cost. 9.3. execution environment. in addition to the system state σ, the remaining gas for computation g, and the accrued substate a, there are several pieces of important information used in the execution environment that the execution agent must provide; these are contained in the tuple i: • ia, the address of the account which owns the code that is executing. • io, the sender address of the transaction that originated this execution. • ip, the price of gas paid by the signer of the transaction that originated this execution. this is defined as the effective gas price p in section 6. • id, the byte array that is the input data to this execution; if the execution agent is a transaction, this would be the transaction data. • is, the address of the account which caused the code to be executing; if the execution agent is a transaction, this would be the transaction sender. • iv, the value, in wei, passed to this account as part of the same procedure as execution; if the execution agent is a transaction, this would be the transaction value. • ib, the byte array that is the machine code to be executed. • ih, the block header of the present block. • ie, the depth of the present message-call or contract-creation (i.e. the number of calls or create(2)s being executed at present). • iw, the permission to make modifications to the state. the execution model defines the function ξ, which can compute the resultant state σ′, the remaining gas g′, the resultant accrued substate a′ and the resultant output, o, given these definitions. for the present context, we will define it as: (143) (σ ′ ,g ′ ,a ′ , o) ≡ ξ(σ,g,a,i) where we will remember that a, the accrued substate, is defined in section 6.1. 9.4. execution overview. we must now define the ξ function. in most practical implementations this will be modelled as an iterative progression of the pair comprising the full system state, σ and the machine state, µ. formally, we define it recursively with a function x. this uses an iterator function o (which defines the result of a single cycle of the state machine) together with functions z which determines if the present state is an exceptional halting state of the machine and h, specifying the output data of the instruction if and only if the present state is a normal halting state of the machine. the empty sequence, denoted (), is not equal to the empty set, denoted ∅; this is important when interpreting the output of h, which evaluates to ∅ when execution is to continue but a series (potentially empty) when execution ethereum: a secure decentralised generalised transaction ledger shanghai version 15 should halt. ξ(σ,g,a,i) ≡ (σ′,µ′g,a ′ , o)(144) (σ ′ ,µ ′ ,a ′ , ..., o) ≡ x ( (σ,µ,a,i) ) (145) µg ≡ g(146) µpc ≡ 0(147) µm ≡ (0, 0, ...)(148) µi ≡ 0(149) µs ≡ ()(150) µo ≡ ()(151) (152) x ( (σ,µ,a,i) ) ≡   ( ∅,µ,a,i,∅ ) if z(σ,µ,a,i)( ∅,µ′,a,i, o ) if w = revert o(σ,µ,a,i) · o if o 6= ∅ x ( o(σ,µ,a,i) ) otherwise where o ≡ h(µ,i)(153) (a,b,c,d) ·e ≡ (a,b,c,d,e)(154) µ ′ ≡ µ except:(155) µ ′ g ≡ µg −c(σ,µ,a,i)(156) note that, when we evaluate ξ, we drop the fourth element i′ and extract the remaining gas µ′g from the resultant machine state µ′. x is thus cycled (recursively here, but implementations are generally expected to use a simple iterative loop) until either z becomes true indicating that the present state is exceptional and that the machine must be halted and any changes discarded or until h becomes a series (rather than the empty set) indicating that the machine has reached a controlled halt. 9.4.1. machine state. the machine state µ is defined as the tuple (g,pc, m, i, s, o) which are the gas available, the program counter pc ∈ n256 , the memory contents, the active number of words in memory (counting continuously from position 0), the stack contents, and the returndata buffer. the memory contents µm are a series of zeroes of size 2256. for the ease of reading, the instruction mnemonics, written in small-caps (e.g. add), should be interpreted as their numeric equivalents; the full table of instructions and their specifics is given in appendix h. for the purposes of defining z, h and o, we define w as the current operation to be executed: (157) w ≡ { ib[µpc] if µpc < ‖ib‖ stop otherwise we also assume the fixed amounts of δ and α, specifying the stack items removed and added, both subscriptable on the instruction and an instruction cost function c evaluating to the full cost, in gas, of executing the given instruction. 9.4.2. exceptional halting. the exceptional halting function z is defined as: (158) z(σ,µ,a,i) ≡ µg < c(σ,µ,a,i) ∨ δw = ∅ ∨ ‖µs‖ < δw ∨ (w = jump ∧ µs[0] /∈ d(ib)) ∨ (w = jumpi ∧ µs[1] 6= 0 ∧ µs[0] /∈ d(ib)) ∨ (w = returndatacopy ∧ µs[1] + µs[2] > ‖µo‖) ∨ ‖µs‖− δw + αw > 1024 ∨ (¬iw ∧ w(w,µ)) ∨ (w = sstore ∧ µg 6 gcallstipend) where (159) w(w,µ) ≡ w ∈{create, create2, sstore, selfdestruct} ∨ log0 ≤ w ∧ w ≤ log4 ∨ w = call ∧ µs[2] 6= 0 this states that the execution is in an exceptional halting state if there is insufficient gas, if the instruction is invalid (and therefore its δ subscript is undefined), if there are insufficient stack items, if a jump/jumpi destination is invalid, the new stack size would be larger than 1024 or state modification is attempted during a static call. the astute reader will realise that this implies that no instruction can, through its execution, cause an exceptional halt. also, the execution is in an exceptional halting state if the gas left prior to executing an sstore instruction is less than or equal to the call stipend gcallstipend – see eip-2200 by tang [2019] for more information. 9.4.3. jump destination validity. we previously used d as the function to determine the set of valid jump destinations given the code that is being run. we define this as any position in the code occupied by a jumpdest instruction. all such positions must be on valid instruction boundaries, rather than sitting in the data portion of push operations and must appear within the explicitly defined portion of the code (rather than in the implicitly defined stop operations that trail it). formally: (160) d(c) ≡ dj(c, 0) where: (161) dj(c, i) ≡   {} if i > ‖c‖ {i}∪dj(c,n(i, c[i])) if c[i] = jumpdest dj(c,n(i, c[i])) otherwise where n is the next valid instruction position in the code, skipping the data of a push instruction, if any: (162) n(i,w) ≡   i + w − push1 + 2 if w ∈ [push1, push32] i + 1 otherwise ethereum: a secure decentralised generalised transaction ledger shanghai version 16 9.4.4. normal halting. the normal halting function h is defined: (163) h(µ,i) ≡   hreturn(µ) if w ∈{return, revert} () if w ∈{stop, selfdestruct} ∅ otherwise the data-returning halt operations, return and revert, have a special function hreturn. note also the difference between the empty sequence and the empty set as discussed here. 9.5. the execution cycle. stack items are added or removed from the left-most, lower-indexed portion of the series; all other items remain unchanged: o ( (σ,µ,a,i) ) ≡ (σ′,µ′,a′,i)(164) ∆ ≡ αw − δw(165) ‖µ′s‖ ≡ ‖µs‖ + ∆(166) ∀x ∈ [αw,‖µ′s‖) : µ ′ s[x] ≡ µs[x− ∆](167) the gas is reduced by the instruction’s gas cost and for most instructions, the program counter increments on each cycle, for the three exceptions, we assume a function j, subscripted by one of two instructions, which evaluates to the according value: µ ′ g ≡ µg −c(σ,µ,a,i)(168) µ ′ pc ≡   jjump(µ) if w = jump jjumpi(µ) if w = jumpi n(µpc,w) otherwise (169) in general, we assume the memory, accrued substate and system state do not change: µ ′ m ≡ µm(170) µ ′ i ≡ µi(171) a ′ ≡ a(172) σ ′ ≡ σ(173) however, instructions do typically alter one or several components of these values. altered components listed by instruction are noted in appendix h, alongside values for α and δ and a formal description of the gas requirements. 10. transition to proof of stake the paris hard fork changed the underlying consensus mechanism of ethereum from proof of work to proof of stake. unlike all previous hard forks of ethereum, paris was not defined to occur at a particular block height, but rather after a specified terminal total difficulty was reached. total difficulty was used instead of block height to avoid a scenario in which a minority of hash power could create a malicious fork that could race to satisfy the block height requirement and claim the first proof of stake block. thus the terminal block, the last proof of work block before the paris fork takes effect, is defined as having: bt > 58750000000000000000000(174) p(bh)t < 58750000000000000000000(175) where bt is the total difficulty of block b and p(bh)t is the total difficulty of its parent. total difficulty for a proof of work (pre-paris ) block b is defined recursively as: bt ≡ p(bh)t + hd(176) where hd is the difficulty of the current block b. upon reaching the terminal block, new blocks are processed by the beacon chain. 10.1. post-paris updates. because the beacon chain generate a new slot every 12 seconds, post-paris updates can be scheduled to occur at a specific timestamp. at the execution layer, the update will then happen in the first produced block after the scheduled timestamp. for example the shanghai hard fork was scheduled to occur at 2023-04-12 10:27:35 utc, on epoch 194,048. validators failed to propose a block during the two first slots of this epoch, but at slot 6,209,538 a validator finally proposed the block 17,034,870, marking the transition to shanghai on the execution layer. 11. blocktree to blockchain prior to the transition to proof of stake at the paris hard fork, the canonical blockchain was defined as the block path with the greatest total difficulty, defined in section 10 as bt. after reaching the terminal block described in section 10, the greatest total difficulty rule must be removed in favor of a rule known as lmd ghost.12 note that in order to determine what blocks comprise the canonical ethereum blockchain after paris, one must have additional information from the beacon chain, which is not described herein. we denote events emitted by the beacon chain with the prefix pos . on each occurrence of a pos forkchoice updated event, starting with the first at the transition block described in section 10, the canonical chain is defined as the chain beginning with the genesis block and ending at the block nominated by the event as the head of the chain. the head of the chain should be updated if and only if a pos forkchoice updated is emitted, in which case the head should set to the block specified by the event. no optimistic updates to the head of the chain should be made. the pos forkchoice updated event additionally references a finalized block. the most recent finalized block should be set to this block. the canonical blockchain must also contain a block with the hash and number of the terminal block defined in section 10. 12. block finalisation the process of finalising a block involves three stages: (1) executing withdrawals; (2) validate transactions; (3) verify state. 12lmd ghost comprises two acronyms, “latest message driven”, and “greedy heaviest-observed sub-tree”. ethereum: a secure decentralised generalised transaction ledger shanghai version 17 12.1. executing withdrawals. after processing the block’s transactions, the withdrawals are executed. a withdrawal is simply an increase of the recipient account’s balance of the specified gwei amount. no other balances are decreased, a withdrawal is not a transfer but a creation of funds. a withdrawal operation cannot fail and has no gas cost. we define the function e as the withdrawal state transition function: e(σw,w) ≡ σw+1(177) σw+1 ≡ σw except:(178) σw+1[wr]b ≡ σw[wr]b + (wa × 109)(179) finally, we define k, the block-level state transition function for withdrawals: (180) k(σ,b) ≡ e(e(σ,w0),w1)... 12.2. transaction validation. the given gasused must correspond faithfully to the transactions listed: bhg, the total gas used in the block, must be equal to the accumulated gas used according to the final transaction: (181) bhg = `(r)u 12.3. state validation. we may now define the function, γ, that maps a block b to its initiation state: (182) γ(b) ≡ { σ0 if p(bh) = ∅ σi : trie(ls(σi)) = p(bh)hr otherwise here, trie(ls(σi)) means the hash of the root node of a trie of state σi; it is assumed that implementations will store this in the state database, which is trivial and efficient since the trie is by nature an immutable data structure. and finally we define φ, the block transition function, which maps an incomplete block b to a complete block b′: φ(b) ≡ b′ : b′ = b except:(183) b ′ hr = trie(ls(k(π(γ(b),b),b)))(184) as specified at the beginning of the present work, π is the state-transition function, which is defined in terms of υ, the transaction-evaluation function. as previously detailed, r[n]z, r[n]l and r[n]u are the nth corresponding status code, logs and cumulative gas used after each transaction (r[n]b, the fourth component in the tuple, has already been defined in terms of the logs). we also define the nth state σ[n], which is defined simply as the state resulting from applying the corresponding transaction to the state resulting from the previous transaction (or the block’s initial state in the case of the first such transaction): (185) σ[n] = { γ(b) if n < 0 υ(σ[n− 1],bt[n]) otherwise in the case of br[n]u, we take a similar approach defining each item as the gas used in evaluating the corresponding transaction summed with the previous item (or zero, if it is the first), giving us a running total: (186) r[n]u =   0 if n < 0 υg(σ[n− 1],bt[n]) +r[n− 1]u otherwise for r[n]l, we utilise the υ l function that we conveniently defined in the transaction execution function. (187) r[n]l = υ l (σ[n− 1],bt[n]) we define r[n]z in a similar manner. (188) r[n]z = υ z (σ[n− 1],bt[n]) finally, we define π as the final transaction’s resultant state, `(σ): (189) π(σ,b) ≡ `(σ) thus the complete block-transition mechanism (before consensus) is defined. 13. implementing contracts there are several patterns of contracts engineering that allow particular useful behaviours; two of these that we will briefly discuss are data feeds and random numbers. 13.1. data feeds. a data feed contract is one which provides a single service: it gives access to information from the external world within ethereum. the accuracy and timeliness of this information is not guaranteed and it is the task of a secondary contract author—the contract that utilises the data feed—to determine how much trust can be placed in any single data feed. the general pattern involves a single contract within ethereum which, when given a message call, replies with some timely information concerning an external phenomenon. an example might be the local temperature of new york city. this would be implemented as a contract that returned that value of some known point in storage. of course this point in storage must be maintained with the correct such temperature, and thus the second part of the pattern would be for an external server to run an ethereum node, and immediately on discovery of a new block, creates a new valid transaction, sent to the contract, updating said value in storage. the contract’s code would accept such updates only from the identity contained on said server. 13.2. random numbers. providing random numbers within a deterministic system is, naturally, an impossible task. however, we can approximate with pseudo-random numbers by utilising data which is generally unknowable at the time of transacting. such data might include the block’s hash and the block’s beneficiary address. in order to make it hard for malicious validators to control those values, one should use the blockhash operation in order to use hashes of the previous 256 blocks as pseudo-random numbers. for a series of such numbers, a trivial solution would be to add some constant amount and hashing the result. 14. future directions the state database won’t be forced to maintain all past state trie structures into the future. it should maintain an age for each node and eventually discard nodes that are neither recent enough nor checkpoints. checkpoints, or a set of nodes in the database that allow a particular block’s state trie to be traversed, could be used to place a maximum limit on the amount of computation needed in order to retrieve any state throughout the blockchain. blockchain consolidation could be used in order to reduce the amount of blocks a client would need to download ethereum: a secure decentralised generalised transaction ledger shanghai version 18 to act as a full node. a compressed archive of the trie structure at given points in time (perhaps one in every 10,000th block) could be maintained by the peer network, effectively recasting the genesis block. this would reduce the amount to be downloaded to a single archive plus a hard maximum limit of blocks. finally, blockchain compression could perhaps be conducted: nodes in state trie that haven’t sent/received a transaction in some constant amount of blocks could be thrown out, reducing both ether-leakage and the growth of the state database. 15. conclusion we have introduced, discussed and formally defined the protocol of ethereum. through this protocol the reader may implement a node on the ethereum network and join others in a decentralised secure social operating system. contracts may be authored in order to algorithmically specify and autonomously enforce rules of interaction. 16. acknowledgements many thanks to aeron buchanan for authoring the homestead revisions, christoph jentzsch for authoring the ethash algorithm and yoichi hirai for doing most of the eip-150 changes. important maintenance, useful corrections and suggestions were provided by a number of others from the ethereum dev organisation and ethereum community at large including gustav simonsson, pawe l bylica, jutta steiner, nick savers, viktor trón, marko simovic, giacomo tazzari and, of course, vitalik buterin. 17. availability the source of this paper is maintained at https: //github.com/ethereum/yellowpaper/. an autogenerated pdf is located at https://ethereum.github. io/yellowpaper/paper.pdf. references jacob aron. bitcoin software finds new life. new scientist, 213(2847):20, 2012. url http://www.sciencedirect. com/science/article/pii/s0262407912601055. adam back. hashcash amortizable publicly auditable cost-functions, 2002. url http://www.hashcash.org/ papers/amortizable.pdf. alex beregszaszi, pawe l bylica, andrei maiboroda, alexey akhunov, christian reitwiessner, and martin swende. eip-3541: reject new contract code starting with the 0xef byte, march 2021. url https://eips.ethereum. org/eips/eip-3541. guido bertoni, joan daemen, michal peeters, and gilles van assche. the keccak sha-3 submission, 2011. url https://keccak.team/files/ keccak-submission-3.pdf. roman boutellier and mareike heinzen. pirates, pioneers, innovators and imitators. in growth through innovation, pages 85–96. springer, 2014. url https: //www.springer.com/gb/book/9783319040158. vitalik buterin. ethereum: a next-generation smart contract and decentralized application platform, 2013. url https://github.com/ethereum/wiki/ wiki/white-paper. vitalik buterin. eip-2: homestead hard-fork changes, 2015. url https://eips.ethereum.org/eips/eip-2. vitalik buterin. eip-155: simple replay attack protection, october 2016. url https://eips.ethereum.org/ eips/eip-155. vitalik buterin. eip-1014: skinny create2, april 2018. url https://eips.ethereum.org/eips/eip-1014. vitalik buterin and martin swende. eip-2929: gas cost increases for state access opcodes, september 2020a. url https://eips.ethereum.org/eips/eip-2929. vitalik buterin and martin swende. eip-2930: optional access lists, august 2020b. url https://eips.ethereum. org/eips/eip-2930. vitalik buterin and martin swende. eip-3529: reduction in refunds, april 2021. url https://eips.ethereum. org/eips/eip-3529. vitalik buterin, eric conner, rick dudley, matthew slipper, ian norden, and abdelhamid bakhta. eip1559: fee market change for eth 1.0 chain, 2019. url https://eips.ethereum.org/eips/eip-1559. nicolas t. courtois, marek grajek, and rahul naik. optimizing sha256 in bitcoin mining, pages 131– 144. springer berlin heidelberg, berlin, heidelberg, 2014. isbn 978-3-662-44893-9. doi: 10. 1007/978-3-662-44893-9 12. url https://doi.org/10. 1007/978-3-662-44893-9_12. b.a. davey and h.a. priestley. introduction to lattices and order. 2nd ed. cambridge: cambridge university press, 2nd ed. edition, 2002. isbn 0-521-78451-4/pbk. cynthia dwork and moni naor. pricing via processing or combatting junk mail. in in 12th annual international cryptology conference, pages 139– 147, 1992. url http://www.wisdom.weizmann.ac.il/ ~naor/papers/pvp.pdf. dankrad feist, dmitry khovratovich, and marius van der wijden. eip-3607: reject transactions from senders with deployed code, june 2021. url https://eips. ethereum.org/eips/eip-3607. phong vo glenn fowler, landon curt noll. fowlernollvo hash function, 1991. url http://www.isthe.com/ chongo/tech/comp/fnv/index.html. nils gura, arun patel, arvinderpal wander, hans eberle, and sheueling chang shantz. comparing elliptic curve cryptography and rsa on 8-bit cpus. in cryptographic hardware and embedded systems-ches 2004, pages 119–132. springer, 2004. url https://www.iacr.org/ archive/ches2004/31560117/31560117.pdf. tjaden hess, matt luongo, piotr dyraga, and james hancock. eip-152: add blake2 compression function ‘f‘ precompile, october 2016. url https://eips. ethereum.org/eips/eip-152. don johnson, alfred menezes, and scott vanstone. the elliptic curve digital signature algorithm (ecdsa), 2001. url https: //web.archive.org/web/20170921160141/http:// cs.ucsb.edu/~koc/ccs130h/notes/ecdsa-cert.pdf. accessed 21 september 2017, but the original link was inaccessible on 19 october 2017. refer to section 6.2 for ecdsapubkey, and section 7 for ecdsasign and ecdsarecover. sergio demian lerner. strict memory hard hashing functions, 2014. url http://www.hashcash.org/papers/ memohash.pdf. https://github.com/ethereum/yellowpaper/ https://github.com/ethereum/yellowpaper/ https://ethereum.github.io/yellowpaper/paper.pdf https://ethereum.github.io/yellowpaper/paper.pdf http://www.sciencedirect.com/science/article/pii/s0262407912601055 http://www.sciencedirect.com/science/article/pii/s0262407912601055 http://www.hashcash.org/papers/amortizable.pdf http://www.hashcash.org/papers/amortizable.pdf https://eips.ethereum.org/eips/eip-3541 https://eips.ethereum.org/eips/eip-3541 https://keccak.team/files/keccak-submission-3.pdf https://keccak.team/files/keccak-submission-3.pdf https://www.springer.com/gb/book/9783319040158 https://www.springer.com/gb/book/9783319040158 https://github.com/ethereum/wiki/wiki/white-paper https://github.com/ethereum/wiki/wiki/white-paper https://eips.ethereum.org/eips/eip-2 https://eips.ethereum.org/eips/eip-155 https://eips.ethereum.org/eips/eip-155 https://eips.ethereum.org/eips/eip-1014 https://eips.ethereum.org/eips/eip-2929 https://eips.ethereum.org/eips/eip-2930 https://eips.ethereum.org/eips/eip-2930 https://eips.ethereum.org/eips/eip-3529 https://eips.ethereum.org/eips/eip-3529 https://eips.ethereum.org/eips/eip-1559 https://doi.org/10.1007/978-3-662-44893-9_12 https://doi.org/10.1007/978-3-662-44893-9_12 http://www.wisdom.weizmann.ac.il/~naor/papers/pvp.pdf http://www.wisdom.weizmann.ac.il/~naor/papers/pvp.pdf https://eips.ethereum.org/eips/eip-3607 https://eips.ethereum.org/eips/eip-3607 http://www.isthe.com/chongo/tech/comp/fnv/index.html http://www.isthe.com/chongo/tech/comp/fnv/index.html https://www.iacr.org/archive/ches2004/31560117/31560117.pdf https://www.iacr.org/archive/ches2004/31560117/31560117.pdf https://eips.ethereum.org/eips/eip-152 https://eips.ethereum.org/eips/eip-152 https://web.archive.org/web/20170921160141/http://cs.ucsb.edu/~koc/ccs130h/notes/ecdsa-cert.pdf https://web.archive.org/web/20170921160141/http://cs.ucsb.edu/~koc/ccs130h/notes/ecdsa-cert.pdf https://web.archive.org/web/20170921160141/http://cs.ucsb.edu/~koc/ccs130h/notes/ecdsa-cert.pdf http://www.hashcash.org/papers/memohash.pdf http://www.hashcash.org/papers/memohash.pdf ethereum: a secure decentralised generalised transaction ledger shanghai version 19 mark miller. the future of law. in paper delivered at the extro 3 conference (august 9), 1997. url https://drive.google.com/file/d/ 0bw0vxjkbgypms0j2vgiywwlocms/edit?usp=sharing. satoshi nakamoto. bitcoin: a peer-to-peer electronic cash system, 2008. url http://www.bitcoin.org/bitcoin. pdf. meni rosenfeld, yoni assia, vitalik buterin, m liorhakilior, oded leiba, assaf shomer, and eliran zach. colored coins protocol specification, 2012. url https://github.com/colored-coins/ colored-coins-protocol-specification. markku-juhani saarinen and jean-philippe aumasson. rfc 7693: the blake2 cryptographic hash and message authentication code (mac), november 2015. url https://tools.ietf.org/html/rfc7693. simon sprankel. technical basis of digital currencies, 2013. url http://www.coderblog.de/wp-content/ uploads/technical-basis-of-digital-currencies. pdf. nick szabo. formalizing and securing relationships on public networks. first monday, 2(9), 1997. url http://firstmonday.org/ojs/index.php/ fm/article/view/548. wei tang. eip-2200: structured definitions for net gas metering, 2019. url https://eips.ethereum.org/eips/ eip-2200. vivek vishnumurthy, sangeeth chandrakumar, and emin gn sirer. karma: a secure economic framework for peer-to-peer resource sharing, 2003. url https:// www.cs.cornell.edu/people/egs/papers/karma.pdf. j. r. willett. mastercoin complete specification, 2013. url https://github.com/mastercoin-msc/spec. micah zoltu. eip-2718: typed transaction envelope, june 2020. url https://eips.ethereum.org/eips/ eip-2718. appendix a. terminology external actor: a person or other entity able to interface to an ethereum node, but external to the world of ethereum. it can interact with ethereum through depositing signed transactions and inspecting the blockchain and associated state. has one (or more) intrinsic accounts. address: a 160-bit code used for identifying accounts. account: accounts have an intrinsic balance and transaction count maintained as part of the ethereum state. they also have some (possibly empty) evm code and a (possibly empty) storage state associated with them. though homogenous, it makes sense to distinguish between two practical types of account: those with empty associated evm code (thus the account balance is controlled, if at all, by some external entity) and those with non-empty associated evm code (thus the account represents an autonomous object). each account has a single address that identifies it. transaction: a piece of data, signed by an external actor. it represents either a message or a new autonomous object. transactions are recorded into each block of the blockchain. autonomous object: a notional object existent only within the hypothetical state of ethereum. has an intrinsic address and thus an associated account; the account will have non-empty associated evm code. incorporated only as the storage state of that account. storage state: the information particular to a given account that is maintained between the times that the account’s associated evm code runs. message: data (as a set of bytes) and value (specified as ether) that is passed between two accounts, either through the deterministic operation of an autonomous object or the cryptographically secure signature of the transaction. message call: the act of passing a message from one account to another. if the destination account is associated with non-empty evm code, then the vm will be started with the state of said object and the message acted upon. if the message sender is an autonomous object, then the call passes any data returned from the vm operation. gas: the fundamental network cost unit. paid for exclusively by ether (as of poc-4), which is converted freely to and from gas as required. gas does not exist outside of the internal ethereum computation engine; its price is set by the transaction and validators are free to ignore transactions whose gas price is too low. contract: informal term used to mean both a piece of evm code that may be associated with an account or an autonomous object. object: synonym for autonomous object. app: an end-user-visible application hosted in the ethereum browser. ethereum browser: (aka ethereum reference client) a cross-platform gui of an interface similar to a simplified browser (a la chrome) that is able to host sandboxed applications whose backend is purely on the ethereum protocol. ethereum virtual machine: (aka evm) the virtual machine that forms the key part of the execution model for an account’s associated evm code. ethereum runtime environment: (aka ere) the environment which is provided to an autonomous object executing in the evm. includes the evm but also the structure of the world state on which the evm relies for certain i/o instructions including call & create. evm code: the bytecode that the evm can natively execute. used to formally specify the meaning and ramifications of a message to an account. evm assembly: the human-readable form of evm-code. https://drive.google.com/file/d/0bw0vxjkbgypms0j2vgiywwlocms/edit?usp=sharing https://drive.google.com/file/d/0bw0vxjkbgypms0j2vgiywwlocms/edit?usp=sharing http://www.bitcoin.org/bitcoin.pdf http://www.bitcoin.org/bitcoin.pdf https://github.com/colored-coins/colored-coins-protocol-specification https://github.com/colored-coins/colored-coins-protocol-specification https://tools.ietf.org/html/rfc7693 http://www.coderblog.de/wp-content/uploads/technical-basis-of-digital-currencies.pdf http://www.coderblog.de/wp-content/uploads/technical-basis-of-digital-currencies.pdf http://www.coderblog.de/wp-content/uploads/technical-basis-of-digital-currencies.pdf http://firstmonday.org/ojs/index.php/fm/article/view/548 http://firstmonday.org/ojs/index.php/fm/article/view/548 https://eips.ethereum.org/eips/eip-2200 https://eips.ethereum.org/eips/eip-2200 https://www.cs.cornell.edu/people/egs/papers/karma.pdf https://www.cs.cornell.edu/people/egs/papers/karma.pdf https://github.com/mastercoin-msc/spec https://eips.ethereum.org/eips/eip-2718 https://eips.ethereum.org/eips/eip-2718 ethereum: a secure decentralised generalised transaction ledger shanghai version 20 lll: the lisp-like low-level language, a human-writable language used for authoring simple contracts and general low-level language toolkit for trans-compiling to. appendix b. recursive length prefix this is a serialisation method for encoding arbitrarily structured binary data (byte arrays). we define the set of possible structures t: t ≡ l]b(190) l ≡ {t : t = (t[0], t[1], ...) ∧ ∀n < ‖t‖ : t[n] ∈ t}(191) b ≡ {b : b = (b[0], b[1], ...) ∧ ∀n < ‖b‖ : b[n] ∈ o}(192) where o is the set of (8-bit) bytes. thus b is the set of all sequences of bytes (otherwise known as byte arrays, and a leaf if imagined as a tree), l is the set of all tree-like (sub-)structures that are not a single leaf (a branch node if imagined as a tree) and t is the set of all byte arrays and such structural sequences. the disjoint union ] is needed only to distinguish the empty byte array () ∈ b from the empty list () ∈ l, which are encoded differently as defined below; as common, we will abuse notation and leave the disjoint union indices implicit, inferable from context. we define the rlp function as rlp through two sub-functions, the first handling the instance when the value is a byte array, the second when it is a sequence of further values: (193) rlp(x) ≡ { rb(x) if x ∈ b rl(x) otherwise if the value to be serialised is a byte array, the rlp serialisation takes one of three forms: • if the byte array contains solely a single byte and that single byte is less than 128, then the input is exactly equal to the output. • if the byte array contains fewer than 56 bytes, then the output is equal to the input prefixed by the byte equal to the length of the byte array plus 128. • otherwise, the output is equal to the input, provided that it contains fewer than 264 bytes, prefixed by the minimal-length byte array which when interpreted as a big-endian integer is equal to the length of the input byte array, which is itself prefixed by the number of bytes required to faithfully encode this length value plus 183. byte arrays containing 264 or more bytes cannot be encoded. this restriction ensures that the first byte of the encoding of a byte array is always below 192, and thus it can be readily distinguished from the encodings of sequences in l. formally, we define rb: rb(x) ≡   x if ‖x‖ = 1 ∧ x[0] < 128 (128 + ‖x‖) · x else if ‖x‖ < 56( 183 + ∥∥be(‖x‖)∥∥) ·be(‖x‖) · x else if ‖x‖ < 264 ∅ otherwise (194) be(x) ≡ (b0,b1, ...) : b0 6= 0 ∧x = ‖b‖−1∑ n=0 bn · 256‖b‖−1−n(195) (x1, ...,xn) · (y1, ...,ym) = (x1, ...,xn,y1, ...,ym)(196) thus be is the function that expands a non-negative integer value to a big-endian byte array of minimal length and the dot operator performs sequence concatenation. if instead, the value to be serialised is a sequence of other items then the rlp serialisation takes one of two forms: • if the concatenated serialisations of each contained item is less than 56 bytes in length, then the output is equal to that concatenation prefixed by the byte equal to the length of this byte array plus 192. • otherwise, the output is equal to the concatenated serialisations, provided that they contain fewer than 264 bytes, prefixed by the minimal-length byte array which when interpreted as a big-endian integer is equal to the length of the concatenated serialisations byte array, which is itself prefixed by the number of bytes required to faithfully encode this length value plus 247. sequences whose concatenated serialized items contain 264 or more bytes cannot be encoded. this restriction ensures that the first byte of the encoding does not exceed 255 (otherwise it would not be a byte). thus we finish by formally defining rl: rl(x) ≡   (192 + ‖s(x)‖) ·s(x) if s(x) 6= ∅∧‖s(x)‖ < 56( 247 + ∥∥be(‖s(x)‖)∥∥) ·be(‖s(x)‖) ·s(x) else if s(x) 6= ∅∧‖s(x)‖ < 264 ∅ otherwise (197) s(x) ≡ { rlp(x[0]) ·rlp(x[1]) · ... if ∀i : rlp(x[i]) 6= ∅ ∅ otherwise (198) ethereum: a secure decentralised generalised transaction ledger shanghai version 21 if rlp is used to encode a scalar, defined only as a non-negative integer (in n, or in nx for any x), it must be encoded as the shortest byte array whose big-endian interpretation is the scalar. thus the rlp of some non-negative integer i is defined as: (199) rlp(i : i ∈ n) ≡ rlp(be(i)) when interpreting rlp data, if an expected fragment is decoded as a scalar and leading zeroes are found in the byte sequence, clients are required to consider it non-canonical and treat it in the same manner as otherwise invalid rlp data, dismissing it completely. there is no specific canonical encoding format for signed or floating-point values. appendix c. hex-prefix encoding hex-prefix encoding is an efficient method of encoding an arbitrary number of nibbles as a byte array. it is able to store an additional flag which, when used in the context of the trie (the only context in which it is used), disambiguates between node types. it is defined as the function hp which maps from a sequence of nibbles (represented by the set y) together with a boolean value to a sequence of bytes (represented by the set b): hp(x, t) : x ∈ y ≡ { (16f(t), 16x[0] + x[1], 16x[2] + x[3], ...) if ‖x‖ is even (16(f(t) + 1) + x[0], 16x[1] + x[2], 16x[3] + x[4], ...) otherwise (200) f(t) ≡ { 2 if t 6= 0 0 otherwise (201) thus the high nibble of the first byte contains two flags; the lowest bit encoding the oddness of the length and the second-lowest encoding the flag t. the low nibble of the first byte is zero in the case of an even number of nibbles and the first nibble in the case of an odd number. all remaining nibbles (now an even number) fit properly into the remaining bytes. appendix d. modified merkle patricia tree the modified merkle patricia tree (trie) provides a persistent data structure to map between arbitrary-length binary data (byte arrays). it is defined in terms of a mutable data structure to map between 256-bit binary fragments and arbitrary-length binary data, typically implemented as a database. the core of the trie, and its sole requirement in terms of the protocol specification, is to provide a single value that identifies a given set of key-value pairs, which may be either a 32-byte sequence or the empty byte sequence. it is left as an implementation consideration to store and maintain the structure of the trie in a manner that allows effective and efficient realisation of the protocol. formally, we assume the input value i, a set containing pairs of byte sequences with unique keys: (202) i = {(k0 ∈ b, v0 ∈ b), (k1 ∈ b, v1 ∈ b), ...} when considering such a sequence, we use the common numeric subscript notation to refer to a tuple’s key or value, thus: (203) ∀i ∈ i : i ≡ (i0,i1) any series of bytes may also trivially be viewed as a series of nibbles, given an endian-specific notation; here we assume big-endian. thus: y(i) = {(k′0 ∈ y, v0 ∈ b), (k ′ 1 ∈ y, v1 ∈ b), ...}(204) ∀n : ∀i < 2‖kn‖ : k′n[i] ≡ { bkn[i÷ 2] ÷ 16c if i is even kn[bi÷ 2c] mod 16 otherwise (205) we define the function trie, which evaluates to the root of the trie that represents this set when encoded in this structure: (206) trie(i) ≡ kec ( rlp(c(i, 0)) ) we also assume a function n, the trie’s node cap function. when composing a node, we use rlp to encode the structure. as a means of reducing storage complexity, we store nodes whose composed rlp is fewer than 32 bytes directly; for those larger we assert prescience of the byte array whose keccak-256 hash evaluates to our reference. thus we define in terms of c, the node composition function: (207) n(i, i) ≡   () ∈ b if i = ∅ c(i, i) if ‖rlp ( c(i, i) ) ‖ < 32 kec ( rlp(c(i, i)) ) otherwise in a manner similar to a radix tree, when the trie is traversed from root to leaf, one may build a single key-value pair. the key is accumulated through the traversal, acquiring a single nibble from each branch node (just as with a radix tree). unlike a radix tree, in the case of multiple keys sharing the same prefix or in the case of a single key having a unique ethereum: a secure decentralised generalised transaction ledger shanghai version 22 suffix, two optimising nodes are provided. thus while traversing, one may potentially acquire multiple nibbles from each of the other two node types, extension and leaf. there are three kinds of nodes in the trie: leaf: a two-item structure whose first item corresponds to the nibbles in the key not already accounted for by the accumulation of keys and branches traversed from the root. the hex-prefix encoding method is used and the second parameter to the function is required to be 1. extension: a two-item structure whose first item corresponds to a series of nibbles of size greater than one that are shared by at least two distinct keys past the accumulation of the keys of nibbles and the keys of branches as traversed from the root. the hex-prefix encoding method is used and the second parameter to the function is required to be 0. branch: a 17-item structure whose first sixteen items correspond to each of the sixteen possible nibble values for the keys at this point in their traversal. the 17th item is used in the case of this being a terminator node and thus a key being ended at this point in its traversal. a branch is then only used when necessary; no branch nodes may exist that contain only a single non-zero entry. we may formally define this structure with the structural composition function c: (208) c(i, i) ≡   ( hp(i0[i..(‖i0‖− 1)], 1),i1 ) if ‖i‖ = 1 where ∃i : i ∈ i( hp(i0[i..(j − 1)], 0),n(i,j) ) if i 6= j where j = max{x : ∃l : ‖l‖ = x∧∀i ∈ i : i0[0..(x− 1)] = l} (u(0),u(1), ...,u(15),v) otherwise where u(j) ≡ n({i : i ∈ i∧ i0[i] = j}, i + 1) v = { i1 if ∃i : i ∈ i∧‖i0‖ = i () ∈ b otherwise d.1. trie database. thus no explicit assumptions are made concerning what data is stored and what is not, since that is an implementation-specific consideration; we simply define the identity function mapping the key-value set i to a 32-byte hash and assert that only a single such hash exists for any i, which though not strictly true is accurate within acceptable precision given the keccak hash’s collision resistance. in reality, a sensible implementation will not fully recompute the trie root hash for each set. a reasonable implementation will maintain a database of nodes determined from the computation of various tries or, more formally, it will memoise the function c. this strategy uses the nature of the trie to both easily recall the contents of any previous key-value set and to store multiple such sets in a very efficient manner. due to the dependency relationship, merkle-proofs may be constructed with an o(log n) space requirement that can demonstrate a particular leaf must exist within a trie of a given root hash. appendix e. precompiled contracts for each precompiled contract, we make use of a template function, ξpre, which implements the out-of-gas checking. (209) ξpre(σ,g,a,i) ≡ { (∅, 0,a, ()) if g < gr (σ,g −gr,a, o) otherwise the precompiled contracts each use these definitions and provide specifications for the o (the output data) and gr, the gas requirements. we define ξecrec as a precompiled contract for the elliptic curve digital signature algorithm (ecdsa) public key recovery function (ecrecover). see appendix f for the definition of the function ecdsarecover and the constant secp256k1n. we also define d to be the input data, well-defined for an infinite length by appending zeroes as required. in the case of an invalid signature, we return no output. ξecrec ≡ ξpre where:(210) gr = 3000(211) ‖o‖ =   0 if v /∈{27, 28} ∨ r = 0 ∨ r ≥ secp256k1n ∨ s = 0 ∨ s ≥ secp256k1n 0 if ecdsarecover(h,v − 27,r,s) = ∅ 32 otherwise (212) if ‖o‖ = 32 :(213) o[0..11] = 0(214) o[12..31] = kec ( ecdsarecover(h,v − 27,r,s) ) [12..31] where:(215) d[0..(‖id‖− 1)] = id(216) d[‖id‖..] = (0, 0, ...)(217) h = d[0..31](218) v = d[32..63](219) r = d[64..95](220) s = d[96..127](221) ethereum: a secure decentralised generalised transaction ledger shanghai version 23 we define ξsha256 and ξrip160 as precompiled contracts implementing the sha2-256 and ripemd-160 hash functions respectively. their gas usage is dependent on the input data size, a factor rounded up to the nearest number of words. ξsha256 ≡ ξpre where:(222) gr = 60 + 12 ⌈‖id‖ 32 ⌉ (223) o[0..31] = sha256(id)(224) ξrip160 ≡ ξpre where:(225) gr = 600 + 120 ⌈‖id‖ 32 ⌉ (226) o[0..11] = 0(227) o[12..31] = ripemd160(id)(228) for the purposes here, we assume we have well-defined standard cryptographic functions for ripemd-160 and sha2-256 of the form: sha256(i ∈ b) ≡ o ∈ b32(229) ripemd160(i ∈ b) ≡ o ∈ b20(230) the fourth contract, the identity function ξid simply defines the output as the input: ξid ≡ ξpre where:(231) gr = 15 + 3 ⌈‖id‖ 32 ⌉ (232) o = id(233) the fifth contract performs arbitrary-precision exponentiation under modulo. here, 00 is taken to be one, and x mod 0 is zero for all x. the first word in the input specifies the number of bytes that the first non-negative integer b occupies. the second word in the input specifies the number of bytes that the second non-negative integer e occupies. the third word in the input specifies the number of bytes that the third non-negative integer m occupies. these three words are followed by b, e and m. the rest of the input is discarded. whenever the input is too short, the missing bytes are considered to be zero. the output is encoded big-endian into the same format as m’s. ξexpmod ≡ ξpre except:(234) gr = max ( 200, ⌊ f ( max(`m,`b) ) max(`′e, 1) gquaddivisor ⌋) (235) gquaddivisor ≡ 3(236) f(x) ≡ ⌈ x 8 ⌉2 (237) ` ′ e =   0 if `e ≤ 32 ∧e = 0 blog2(e)c if `e ≤ 32 ∧e 6= 0 8(`e − 32) + blog2(i[(96 + `b)..(127 + `b)])c if 32 < `e ∧ i[(96 + `b)..(127 + `b)] 6= 0 8(`e − 32) otherwise (238) o = ( b e mod m ) ∈ n8`m(239) `b ≡ i[0..31](240) `e ≡ i[32..63](241) `m ≡ i[64..95](242) b ≡ i[96..(95 + `b)](243) e ≡ i[(96 + `b)..(95 + `b + `e)](244) m ≡ i[(96 + `b + `e)..(95 + `b + `e + `m)](245) i[x] ≡ { id[x] if x < ‖id‖ 0 otherwise (246) e.1. zksnark related precompiled contracts. we choose two numbers, both of which are prime. p ≡ 21888242871839275222246405745257275088696311157297823662689037894645226208583(247) q ≡ 21888242871839275222246405745257275088548364400416034343698204186575808495617(248) since p is a prime number, {0, 1, . . . ,p− 1} forms a field with addition and multiplication modulo p. we call this field fp. ethereum: a secure decentralised generalised transaction ledger shanghai version 24 we define a set c1 with (249) c1 ≡{(x,y ) ∈ fp ×fp | y 2 = x3 + 3}∪{(0, 0)} we define a binary operation + on c1 for distinct elements (x1,y1), (x2,y2) with (x1,y1) + (x2,y2) ≡ { (x,y ) if x1 6= x2 (0, 0) otherwise (250) λ ≡ y2 −y1 x2 −x1 x ≡ λ2 −x1 −x2 y ≡ λ(x1 −x) −y1 in the case where (x1,y1) = (x2,y2), we define + on c1 with (x1,y1) + (x2,y2) ≡ { (x,y ) if y1 6= 0 (0, 0) otherwise (251) λ ≡ 3x21 2y1 x ≡ λ2 − 2x1 y ≡ λ(x1 −x) −y1 (c1, +) is known to form a group. we define scalar multiplication · with (252) n ·p ≡ (0, 0) + p + · · · + p︸ ︷︷ ︸ n for a natural number n and a point p in c1. we define p1 to be a point (1, 2) on c1. let g1 be the subgroup of (c1, +) generated by p1. g1 is known to be a cyclic group of order q. for a point p in g1, we define logp1 (p ) to be the smallest natural number n satisfying n ·p1 = p . logp1 (p) is at most q − 1. let fp2 be a field fp[i]/(i 2 + 1). we define a set c2 with (253) c2 ≡{(x,y ) ∈ fp2 ×fp2 | y 2 = x 3 + 3(i + 9) −1}∪{(0, 0)} we define a binary operation + and scalar multiplication · with the same equations (250), (251) and (252). (c2, +) is also known to be a group. we define p2 in c2 with p2 ≡ (11559732032986387107991004021392285783925812861821192530917403151452391805634 × i(254) +10857046999023057135944570762232829481370756359578518086990519993285655852781, 4082367875863433681332203403145435568316851327593401208105741076214120093531 × i +8495653923123431417604973247489272438418190587263600148770280649306958101930) we define g2 to be the subgroup of (c2, +) generated by p2. g2 is known to be the only cyclic group of order q on c2. for a point p in g2, we define logp2 (p) be the smallest natural number n satisfying n ·p2 = p. with this definition, logp2 (p) is at most q − 1. let gt be the multiplicative abelian group underlying fq12 . it is known that a non-degenerate bilinear map e : g1 ×g2 → gt exists. this bilinear map is a type three pairing. there are several such bilinear maps, it does not matter which is chosen to be e. let pt = e(p1,p2), a be a set of k points in g1, and b be a set of k points in g2. it follows from the definition of a pairing that the following are equivalent logp1 (a1) × logp2 (b1) + · · · + logp1 (ak) × logp2 (bk) ≡ 1 mod q(255) k∏ i=0 e (ai,bi) = pt(256) thus the pairing operation provides a method to verify (255). a 32 byte number x ∈ p256 might and might not represent an element of fp. (257) δp(x) ≡ { x if x < p ∅ otherwise ethereum: a secure decentralised generalised transaction ledger shanghai version 25 a 64 byte data x ∈ b512 might and might not represent an element of g1. δ1(x) ≡ { g1 if g1 ∈ g1 ∅ otherwise (258) g1 ≡ { (x,y) if x 6= ∅∧y 6= ∅ ∅ otherwise (259) x ≡ δp(x[0..31])(260) y ≡ δp(x[32..63])(261) a 128 byte data x ∈ b1024 might and might not represent an element of g2. δ2(x) ≡ { g2 if g2 ∈ g2 ∅ otherwise (262) g2 ≡ { ((x0i + y0), (x1i + y1)) if x0 6= ∅∧y0 6= ∅∧x1 6= ∅∧y1 6= ∅ ∅ otherwise (263) x0 ≡ δp(x[0..31])(264) y0 ≡ δp(x[32..63])(265) x1 ≡ δp(x[64..95])(266) y1 ≡ δp(x[96..127])(267) we define ξsnarkv as a precompiled contract which checks if (255) holds, for intended use in zksnark verification. ξsnarkv ≡ ξpre except:(268) ξsnarkv(σ,g,a,i) = (∅, 0,a, ()) if f(269) f ≡ (‖id‖ mod 192 6= 0 ∨ (∃j. aj = ∅∨ bj = ∅))(270) k = ‖id‖ 192 (271) gr = 34000k + 45000(272) o[0..31] ≡ { 0x0000000000000000000000000000000000000000000000000000000000000001 if v ∧¬f 0x0000000000000000000000000000000000000000000000000000000000000000 if ¬v ∧¬f (273) v ≡ (logp1 (a1) × logp2 (b1) + · · · + logp1 (ak) × logp2 (bk) ≡ 1 mod q)(274) a1 ≡ δ1(id[0..63])(275) b1 ≡ δ2(id[64..191])(276) ... ak ≡ δ1(id[(‖id‖− 192)..(‖id‖− 129)])(277) bk ≡ δ2(id[(‖id‖− 128)..(‖id‖− 1)])(278) we define a precompiled contract for addition on g1. ξbn add ≡ ξbn pre except:(279) ξbn add(σ,g,a,i) = (∅, 0,a, ()) if x = ∅∨y = ∅(280) gr = 150(281) o ≡ δ−11 (x + y) where + is the group operation in g1(282) x ≡ δ1 ( īd[0..63] ) (283) y ≡ δ1 ( īd[64..127] ) (284) īd[x] ≡ { id[x] if x < ‖id‖ 0 otherwise (285) we define a precompiled contract for scalar multiplication on g1, where īd is defined in (285). ξbn mul ≡ ξpre except:(286) ξbn mul(σ,g,a,i) = (∅, 0,a, ()) if x = ∅(287) gr = 6000(288) o ≡ δ−11 (n ·x) where · is the scalar multiplication in g1(289) x ≡ δ1 ( īd[0..63] ) (290) n ≡ īd[64..95](291) ethereum: a secure decentralised generalised transaction ledger shanghai version 26 e.2. blake2 precompiled contract. eip-152 by hess et al. [2016] defines ξblake2 f as a precompiled contract implementing the compression function f used in the blake2 cryptographic hashing algorithm. the f compression function is specified in rfc 7693 by saarinen and aumasson [2015]. ξblake2 f ≡ ξpre except:(292) ξblake2 f(σ,g,a,i) = (∅, 0,a, ()) if ‖id‖ 6= 213 ∨f /∈{0, 1}(293) gr = r(294) o ≡ le8(h′0) · ... ·le8(h ′ 7)(295) (h ′ 0, . . . ,h ′ 7) ≡ f(h,m,tlow, thigh,f) with r rounds and w = 64(296) be4(r) ≡ id[0..4](297) le8(h0) ≡ id[4..12](298) . . .(299) le8(h7) ≡ id[60..68](300) le8(m0) ≡ id[68..76](301) . . .(302) le8(m15) ≡ id[188..196](303) le8(tlow) ≡ id[196..204](304) le8(thigh) ≡ id[204..212](305) f ≡ id[212](306) where r ∈ b32, ∀i ∈ 0..7 : hi ∈ b64, ∀i ∈ 0..15 : mi ∈ b64, tlow ∈ b64, thigh ∈ b64, f ∈ b8, bek is the k-byte big-endian representation—compare with(195): (307) bek(x) ≡ (b0,b1, ...,bk−1) : x = k−1∑ n=0 bn · 256k−1−n and lek is the k-byte little-endian representation: (308) lek(x) ≡ (b0,b1, ...,bk−1) : x = k−1∑ n=0 bn · 256n appendix f. signing transactions transactions are signed using recoverable ecdsa signatures. this method utilises the secp-256k1 curve as described by courtois et al. [2014], and is implemented similarly to as described by gura et al. [2004] on p. 9 of 15, para. 3. it is assumed that the sender has a valid private key pr, which is a randomly selected positive integer (represented as a byte array of length 32 in big-endian form) in the range [1,secp256k1n− 1]. we assume the existence of functions ecdsapubkey, ecdsasign and ecdsarecover. these are formally defined in the literature, e.g. by johnson et al. [2001]. ecdsapubkey(pr ∈ b32) ≡ pu ∈ b64(309) ecdsasign(e ∈ b32,pr ∈ b32) ≡ (v ∈ b1,r ∈ b32,s ∈ b32)(310) ecdsarecover(e ∈ b32,v ∈ b1,r ∈ b32,s ∈ b32) ≡ pu ∈ b64(311) where pu is the public key, assumed to be a byte array of size 64 (formed from the concatenation of two positive integers each < 2256), pr is the private key, a byte array of size 32 (or a single positive integer in the aforementioned range) and e is the hash of the transaction, h(t). it is assumed that v is the ‘recovery identifier’. the recovery identifier is a 1 byte value specifying the parity and finiteness of the coordinates of the curve point for which r is the x-value; this value is in the range of [0, 3], however we declare the upper two possibilities, representing infinite values, invalid. the value 0 represents an even y value and 1 represents an odd y value. we declare that an ecdsa signature is invalid unless all the following conditions are true: 0 < r < secp256k1n(312) 0 < s < secp256k1n÷ 2 + 1(313) v ∈{0, 1}(314) where: secp256k1n = 115792089237316195423570985008687907852837564279074904382605163141518161494337(315) note that this restriction on s is more stringent than restriction 212 in the ξecrec precompile; see eip-2 by buterin [2015] for more detail. for a given private key, pr, the ethereum address a(pr) (a 160-bit value) to which it corresponds is defined as the rightmost 160-bits of the keccak-256 hash of the corresponding ecdsa public key: (316) a(pr) = b96..255 ( kec ( ecdsapubkey(pr) )) ethereum: a secure decentralised generalised transaction ledger shanghai version 27 the message hash, h(t), to be signed is the keccak-256 hash of the transaction. four different flavours of signing schemes are available: lx(t) ≡   (tn,tp,tg,tt,tv, p) if tx = 0 ∧tw ∈{27, 28} (tn,tp,tg,tt,tv, p,β, (), ()) if tx = 0 ∧tw ∈{2β + 35, 2β + 36} (tc,tn,tp,tg,tt,tv, p,ta) if tx = 1 (tc,tn,tf,tm,tg,tt,tv, p,ta) if tx = 2 (317) where p ≡ { ti if tt = ∅ td otherwise h(t) ≡ { kec(rlp(lx(t))) if tx = 0 kec(tx ·rlp(lx(t))) otherwise (318) the signed transaction g(t,pr) is defined as: g(t,pr) ≡ t except:(319) (ty,tr,ts) = ecdsasign(h(t),pr)(320) reiterating from previously: tr = r(321) ts = s(322) and tw of legacy transcations is either 27 + ty or 2β + 35 + ty. we may then define the sender function s of the transaction as: s(t) ≡ b96..255 ( kec ( ecdsarecover(h(t),v,tr,ts) )) (323) v ≡   tw − 27 if tx = 0 ∧tw ∈{27, 28} (tw − 35) mod 2 if tx = 0 ∧tw ∈{2β + 35, 2β + 36} ty if tx = 1 ∨tx = 2 (324) the assertion that the sender of a signed transaction equals the address of the signer should be self-evident: (325) ∀t : ∀pr : s(g(t,pr)) ≡ a(pr) ethereum: a secure decentralised generalised transaction ledger shanghai version 28 appendix g. fee schedule the fee schedule g is a tuple of scalar values corresponding to the relative costs, in gas, of a number of abstract operations that a transaction may effect. name value description gzero 0 nothing paid for operations of the set wzero. gjumpdest 1 amount of gas to pay for a jumpdest operation. gbase 2 amount of gas to pay for operations of the set wbase. gverylow 3 amount of gas to pay for operations of the set wverylow. glow 5 amount of gas to pay for operations of the set wlow. gmid 8 amount of gas to pay for operations of the set wmid. ghigh 10 amount of gas to pay for operations of the set whigh. gwarmaccess 100 cost of a warm account or storage access. gaccesslistaddress 2400 cost of warming up an account with the access list. gaccessliststorage 1900 cost of warming up a storage with the access list. gcoldaccountaccess 2600 cost of a cold account access. gcoldsload 2100 cost of a cold storage access. gsset 20000 paid for an sstore operation when the storage value is set to non-zero from zero. gsreset 2900 paid for an sstore operation when the storage value’s zeroness remains unchanged or is set to zero. rsclear 4800 refund given (added into refund counter) when the storage value is set to zero from non-zero. the refund amount is defined as gsreset + gaccessliststorage. gselfdestruct 5000 amount of gas to pay for a selfdestruct operation. gcreate 32000 paid for a create operation. gcodedeposit 200 paid per byte for a create operation to succeed in placing code into state. ginitcodeword 2 paid per word of the initcode at the beginning of a deploy transaction, a create, or a create2 operation. gcallvalue 9000 paid for a non-zero value transfer as part of the call operation. gcallstipend 2300 a stipend for the called contract subtracted from gcallvalue for a non-zero value transfer. gnewaccount 25000 paid for a call or selfdestruct operation which creates an account. gexp 10 partial payment for an exp operation. gexpbyte 50 partial payment when multiplied by the number of bytes in the exponent for the exp operation. gmemory 3 paid for every additional word when expanding memory. gtxcreate 32000 paid by all contract-creating transactions after the homestead transition. gtxdatazero 4 paid for every zero byte of data or code for a transaction. gtxdatanonzero 16 paid for every non-zero byte of data or code for a transaction. gtransaction 21000 paid for every transaction. glog 375 partial payment for a log operation. glogdata 8 paid for each byte in a log operation’s data. glogtopic 375 paid for each topic of a log operation. gkeccak256 30 paid for each keccak256 operation. gkeccak256word 6 paid for each word (rounded up) for input data to a keccak256 operation. gcopy 3 partial payment for *copy operations, multiplied by words copied, rounded up. gblockhash 20 payment for each blockhash operation. appendix h. virtual machine specification when interpreting 256-bit binary values as integers, the representation is big-endian. when a 256-bit machine datum is converted to and from a 160-bit address or hash, the rightwards (low-order for be) 20 bytes are used and the leftmost 12 are discarded or filled with zeroes, thus the integer values (when the bytes are interpreted as big-endian) are equivalent. ethereum: a secure decentralised generalised transaction ledger shanghai version 29 h.1. gas cost. the general gas cost function, c, is defined as: (326) c(σ,µ,a,i) ≡ cmem(µ′i)−cmem(µi) +   csstore(σ,µ,a,i) if w = sstore gexp if w = exp ∧µs[1] = 0 gexp + gexpbyte × (1 + blog256(µs[1])c) if w = exp ∧µs[1] > 0 gverylow + gcopy ×dµs[2] ÷ 32e if w ∈ wcopy caaccess(µs[0] mod 2 160,a) + gcopy ×dµs[3] ÷ 32e if w = extcodecopy caaccess(µs[0] mod 2 160,a) if w ∈ wextaccount glog + glogdata ×µs[1] if w = log0 glog + glogdata ×µs[1] + glogtopic if w = log1 glog + glogdata ×µs[1] + 2glogtopic if w = log2 glog + glogdata ×µs[1] + 3glogtopic if w = log3 glog + glogdata ×µs[1] + 4glogtopic if w = log4 ccall(σ,µ,a) if w ∈ wcall cselfdestruct(σ,µ) if w = selfdestruct gcreate + r(µs[2]) if w = create gcreate + gkeccak256word ×dµs[2] ÷ 32e + r(µs[2]) if w = create2 gkeccak256 + gkeccak256word ×dµs[1] ÷ 32e if w = keccak256 gjumpdest if w = jumpdest csload(µ,a,i) if w = sload gzero if w ∈ wzero gbase if w ∈ wbase gverylow if w ∈ wverylow glow if w ∈ wlow gmid if w ∈ wmid ghigh if w ∈ whigh gblockhash if w = blockhash (327) w ≡ { ib[µpc] if µpc < ‖ib‖ stop otherwise where: (328) cmem(a) ≡ gmemory ·a + ⌊ a2 512 ⌋ (329) caaccess(x,a) ≡ { gwarmaccess if x ∈ aa gcoldaccountaccess otherwise with ccall, cselfdestruct, csload and csstore as specified in the appropriate section below. we define the following subsets of instructions: wzero = {stop, return, revert} wbase = {address, origin, caller, callvalue, calldatasize, codesize, gasprice, coinbase, timestamp, number, prevrandao, gaslimit, chainid, returndatasize, pop, pc, msize, gas, basefee, push0} wverylow = {add, sub, not, lt, gt, slt, sgt, eq, iszero, and, or, xor, byte, shl, shr, sar, calldataload, mload, mstore, mstore8, push1, ..., push32, dup*, swap*} wlow = {mul, div, sdiv, mod, smod, signextend, selfbalance} wmid = {addmod, mulmod, jump} whigh = {jumpi} wcopy = {calldatacopy, codecopy, returndatacopy} wcall = {call, callcode, delegatecall, staticcall} wextaccount = {balance, extcodesize, extcodehash} note the memory cost component, given as the product of gmemory and the maximum of 0 & the ceiling of the number of words in size that the memory must be over the current number of words, µi in order that all accesses reference valid memory whether for read or write. such accesses must be for non-zero number of bytes. referencing a zero length range (e.g. by attempting to pass it as the input range to a call) does not require memory to be extended to the beginning of the range. µ′i is defined as this new maximum number of words of active memory; special cases are given where these two are not equal. ethereum: a secure decentralised generalised transaction ledger shanghai version 30 note also that cmem is the memory cost function (the expansion function being the difference between the cost before and after). it is a polynomial, with the higher-order coefficient divided and floored, and thus linear up to 704b of memory used, after which it costs substantially more. while defining the instruction set, we defined the memory-expansion for range function, m, thus: (330) m(s,f, l) ≡ { s if l = 0 max(s,d(f + l) ÷ 32e) otherwise another useful function is “all but one 64th” function l defined as: (331) l(n) ≡ n−bn/64c h.2. instruction set. as previously specified in section 9, these definitions take place in the final context there. in particular we assume o is the evm state-progression function and define the terms pertaining to the next cycle’s state (σ′,µ′) such that: (332) o(σ,µ,a,i) ≡ (σ′,µ′,a′,i) with exceptions, as noted here given are the various exceptions to the state transition rules given in section 9 specified for each instruction, together with the additional instruction-specific definitions of j and c. for each instruction, also specified is α, the additional items placed on the stack and δ, the items removed from stack, as defined in section 9. ethereum: a secure decentralised generalised transaction ledger shanghai version 31 0s: stop and arithmetic operations all arithmetic is modulo 2256 unless otherwise noted. the zero-th power of zero 00 is defined to be one. value mnemonic δ α description 0x00 stop 0 0 halts execution. 0x01 add 2 1 addition operation. µ′s[0] ≡ µs[0] + µs[1] 0x02 mul 2 1 multiplication operation. µ′s[0] ≡ µs[0] ×µs[1] 0x03 sub 2 1 subtraction operation. µ′s[0] ≡ µs[0] −µs[1] 0x04 div 2 1 integer division operation. µ′s[0] ≡ { 0 if µs[1] = 0 bµs[0] ÷µs[1]c otherwise 0x05 sdiv 2 1 signed integer division operation (truncated). µ′s[0] ≡   0 if µs[1] = 0 −2255 if µs[0] = −2 255 ∧ µs[1] = −1 sgn(µs[0] ÷µs[1])b|µs[0] ÷µs[1]|c otherwise where all values are treated as two’s complement signed 256-bit integers. note the overflow semantic when −2255 is negated. 0x06 mod 2 1 modulo remainder operation. µ′s[0] ≡ { 0 if µs[1] = 0 µs[0] mod µs[1] otherwise 0x07 smod 2 1 signed modulo remainder operation. µ′s[0] ≡ { 0 if µs[1] = 0 sgn(µs[0])(|µs[0]| mod |µs[1]|) otherwise where all values are treated as two’s complement signed 256-bit integers. 0x08 addmod 3 1 modulo addition operation. µ′s[0] ≡ { 0 if µs[2] = 0 (µs[0] + µs[1]) mod µs[2] otherwise all intermediate calculations of this operation are not subject to the 2256 modulo. 0x09 mulmod 3 1 modulo multiplication operation. µ′s[0] ≡ { 0 if µs[2] = 0 (µs[0] ×µs[1]) mod µs[2] otherwise all intermediate calculations of this operation are not subject to the 2256 modulo. 0x0a exp 2 1 exponential operation. µ′s[0] ≡ µs[0] µs[1] 0x0b signextend 2 1 extend length of two’s complement signed integer. ∀i ∈ [0..255] : µ′s[0]i ≡ { µs[1]t if i 6 t where t = 256 − 8(µs[0] + 1) µs[1]i otherwise µs[x]i gives the ith bit (counting from zero) of µs[x] ethereum: a secure decentralised generalised transaction ledger shanghai version 32 10s: comparison & bitwise logic operations value mnemonic δ α description 0x10 lt 2 1 less-than comparison. µ′s[0] ≡ { 1 if µs[0] < µs[1] 0 otherwise 0x11 gt 2 1 greater-than comparison. µ′s[0] ≡ { 1 if µs[0] > µs[1] 0 otherwise 0x12 slt 2 1 signed less-than comparison. µ′s[0] ≡ { 1 if µs[0] < µs[1] 0 otherwise where all values are treated as two’s complement signed 256-bit integers. 0x13 sgt 2 1 signed greater-than comparison. µ′s[0] ≡ { 1 if µs[0] > µs[1] 0 otherwise where all values are treated as two’s complement signed 256-bit integers. 0x14 eq 2 1 equality comparison. µ′s[0] ≡ { 1 if µs[0] = µs[1] 0 otherwise 0x15 iszero 1 1 simple not operator. µ′s[0] ≡ { 1 if µs[0] = 0 0 otherwise 0x16 and 2 1 bitwise and operation. ∀i ∈ [0..255] : µ′s[0]i ≡ µs[0]i ∧µs[1]i 0x17 or 2 1 bitwise or operation. ∀i ∈ [0..255] : µ′s[0]i ≡ µs[0]i ∨µs[1]i 0x18 xor 2 1 bitwise xor operation. ∀i ∈ [0..255] : µ′s[0]i ≡ µs[0]i ⊕µs[1]i 0x19 not 1 1 bitwise not operation. ∀i ∈ [0..255] : µ′s[0]i ≡ { 1 if µs[0]i = 0 0 otherwise 0x1a byte 2 1 retrieve single byte from word. ∀i ∈ [0..255] : µ′s[0]i ≡ { µs[1](i−248+8µs[0]) if i ≥ 248 ∧µs[0] < 32 0 otherwise for the nth byte, we count from the left (i.e. n=0 would be the most significant in big endian). 0x1b shl 2 1 left shift operation. µ′s[0] ≡ (µs[1] × 2 µs[0]) mod 2256 0x1c shr 2 1 logical right shift operation. µ′s[0] ≡bµs[1] ÷ 2 µs[0]c 0x1d sar 2 1 arithmetic (signed) right shift operation. µ′s[0] ≡bµs[1] ÷ 2 µs[0]c where µ′s[0] and µs[1] are treated as two’s complement signed 256-bit integers, while µs[0] is treated as unsigned. 20s: keccak256 value mnemonic δ α description 0x20 keccak256 2 1 compute keccak-256 hash. µ′s[0] ≡ kec(µm[µs[0] . . . (µs[0] + µs[1] − 1)]) µ′i ≡ m(µi,µs[0],µs[1]) ethereum: a secure decentralised generalised transaction ledger shanghai version 33 30s: environmental information value mnemonic δ α description 0x30 address 0 1 get address of currently executing account. µ′s[0] ≡ ia 0x31 balance 1 1 get balance of the given account. µ′s[0] ≡ { σ[µs[0] mod 2 160]b if σ[µs[0] mod 2 160] 6= ∅ 0 otherwise a′a ≡ aa ∪{µs[0] mod 2 160} 0x32 origin 0 1 get execution origination address. µ′s[0] ≡ io this is the sender of original transaction; it is never an account with non-empty associated code. 0x33 caller 0 1 get caller address. µ′s[0] ≡ is this is the address of the account that is directly responsible for this execution. 0x34 callvalue 0 1 get deposited value by the instruction/transaction responsible for this execution. µ′s[0] ≡ iv 0x35 calldataload 1 1 get input data of current environment. µ′s[0] ≡ id[µs[0] . . . (µs[0] + 31)] with id[x] = 0 if x > ‖id‖ this pertains to the input data passed with the message call instruction or transaction. 0x36 calldatasize 0 1 get size of input data in current environment. µ′s[0] ≡‖id‖ this pertains to the input data passed with the message call instruction or transaction. 0x37 calldatacopy 3 0 copy input data in current environment to memory. ∀i ∈{0 . . .µs[2] − 1} : µ ′ m[µs[0] + i] ≡ { id[µs[1] + i] if µs[1] + i < ‖id‖ 0 otherwise the additions in µs[1] + i are not subject to the 2 256 modulo. µ′i ≡ m(µi,µs[0],µs[2]) this pertains to the input data passed with the message call instruction or transaction. 0x38 codesize 0 1 get size of code running in current environment. µ′s[0] ≡‖ib‖ 0x39 codecopy 3 0 copy code running in current environment to memory. ∀i ∈{0 . . .µs[2] − 1} : µ ′ m[µs[0] + i] ≡ { ib[µs[1] + i] if µs[1] + i < ‖ib‖ stop otherwise µ′i ≡ m(µi,µs[0],µs[2]) the additions in µs[1] + i are not subject to the 2 256 modulo. 0x3a gasprice 0 1 get price of gas in current environment. this is the effective gas price defined in section 6. note that as of the london hard fork, this value no longer represents what is received by the validator, but rather just what is paid by the sender. µ′s[0] ≡ ip 0x3b extcodesize 1 1 get size of an account’s code. µ′s[0] ≡ { ‖b‖ if σ[µs[0] mod 2 160] 6= ∅ 0 otherwise where kec(b) ≡ σ[µs[0] mod 2 160]c a′a ≡ aa ∪{µs[0] mod 2 160} 0x3c extcodecopy 4 0 copy an account’s code to memory. ∀i ∈{0 . . .µs[3] − 1} : µ ′ m[µs[1] + i] ≡ { b[µs[2] + i] if µs[2] + i < ‖b‖ stop otherwise where kec(b) ≡ σ[µs[0] mod 2 160]c we assume b ≡ () if σ[µs[0] mod 2 160] = ∅. µ′i ≡ m(µi,µs[1],µs[3]) the additions in µs[2] + i are not subject to the 2 256 modulo. a′a ≡ aa ∪{µs[0] mod 2 160} ethereum: a secure decentralised generalised transaction ledger shanghai version 34 0x3d returndatasize 0 1 get size of output data from the previous call from the current environment. µ′s[0] ≡‖µo‖ 0x3e returndatacopy 3 0 copy output data from the previous call to memory. ∀i ∈{0 . . .µs[2] − 1} : µ ′ m[µs[0] + i] ≡ { µo[µs[1] + i] if µs[1] + i < ‖µo‖ 0 otherwise the additions in µs[1] + i are not subject to the 2 256 modulo. µ′i ≡ m(µi,µs[0],µs[2]) 0x3f extcodehash 1 1 get hash of an account’s code. µ′s[0] ≡ { 0 if dead(σ,µs[0] mod 2 160) σ[µs[0] mod 2 160]c otherwise a′a ≡ aa ∪{µs[0] mod 2 160} 40s: block information value mnemonic δ α description 0x40 blockhash 1 1 get the hash of one of the 256 most recent complete blocks. µ′s[0] ≡ p(ihp,µs[0], 0) where p is the hash of a block of a particular number, up to a maximum age. 0 is left on the stack if the looked for block number is greater than or equal to the current block number or more than 256 blocks behind the current block. p(h,n,a) ≡   0 if n > hi ∨a = 256 ∨h = 0 h if n = hi p(hp,n,a + 1) otherwise and we assert the header h can be determined from its hash h unless h is zero (as is the case for the parent hash of the genesis block). 0x41 coinbase 0 1 get the current block’s beneficiary address. µ′s[0] ≡ ihc 0x42 timestamp 0 1 get the current block’s timestamp. µ′s[0] ≡ ihs 0x43 number 0 1 get the current block’s number. µ′s[0] ≡ ihi 0x44 prevrandao 0 1 get the latest randao mix of the post beacon state of the previous block. µ′s[0] ≡ iha 0x45 gaslimit 0 1 get the current block’s gas limit. µ′s[0] ≡ ihl 0x46 chainid 0 1 get the chain id. µ′s[0] ≡ β 0x47 selfbalance 0 1 get balance of currently executing account. µ′s[0] ≡ σ[ia]b 0x48 basefee 0 1 get the current block’s base fee. µ′s[0] ≡ ihf ethereum: a secure decentralised generalised transaction ledger shanghai version 35 50s: stack, memory, storage and flow operations value mnemonic δ α description 0x50 pop 1 0 remove item from stack. 0x51 mload 1 1 load word from memory. µ′s[0] ≡ µm[µs[0] . . . (µs[0] + 31)] µ′i ≡ max(µi,d(µs[0] + 32) ÷ 32e) the addition in the calculation of µ′i is not subject to the 2 256 modulo. 0x52 mstore 2 0 save word to memory. µ′m[µs[0] . . . (µs[0] + 31)] ≡ µs[1] µ′i ≡ max(µi,d(µs[0] + 32) ÷ 32e) the addition in the calculation of µ′i is not subject to the 2 256 modulo. 0x53 mstore8 2 0 save byte to memory. µ′m[µs[0]] ≡ (µs[1] mod 256) µ′i ≡ max(µi,d(µs[0] + 1) ÷ 32e) the addition in the calculation of µ′i is not subject to the 2 256 modulo. 0x54 sload 1 1 load word from storage. µ′s[0] ≡ σ[ia]s[µs[0]] a′k ≡ ak ∪{(ia,µs[0])} csload(µ,a,i) ≡ { gwarmaccess if (ia,µs[0]) ∈ ak gcoldsload otherwise 0x55 sstore 2 0 save word to storage. σ′[ia]s[µs[0]] ≡ µs[1] a′k ≡ ak ∪{(ia,µs[0])} csstore(σ,µ) and a ′ r are specified by eip-2200 as follows. we remind the reader that the checkpoint (“original”) state σ0 is the state if the current transaction were to revert. let v0 = σ0[ia]s[µs[0]] be the original value of the storage slot. let v = σ[ia]s[µs[0]] be the current value. let v′ = µs[1] be the new value. then: csstore(σ,µ,a,i) ≡ { 0 if (ia,µs[0]) ∈ ak gcoldsload otherwise +   gwarmaccess if v = v ′ ∨ v0 6= v gsset if v 6= v′ ∧ v0 = v ∧ v0 = 0 gsreset if v 6= v′ ∧ v0 = v ∧ v0 6= 0 a′r ≡ ar +   rsclear if v 6= v′ ∧ v0 = v ∧ v′ = 0 rdirtyclear + rdirtyreset if v 6= v′ ∧ v0 6= v 0 otherwise where rdirtyclear ≡   −rsclear if v0 6= 0 ∧ v = 0 rsclear if v0 6= 0 ∧ v′ = 0 0 otherwise rdirtyreset ≡   gsset −gwarmaccess if v0 = v′ ∧ v0 = 0 gsreset −gwarmaccess if v0 = v′ ∧ v0 6= 0 0 otherwise 0x56 jump 1 0 alter the program counter. jjump(µ) ≡ µs[0] this has the effect of writing said value to µpc. see section 9. 0x57 jumpi 2 0 conditionally alter the program counter. jjumpi(µ) ≡ { µs[0] if µs[1] 6= 0 µpc + 1 otherwise this has the effect of writing said value to µpc. see section 9. 0x58 pc 0 1 get the value of the program counter prior to the increment corresponding to this instruction. µ′s[0] ≡ µpc ethereum: a secure decentralised generalised transaction ledger shanghai version 36 0x59 msize 0 1 get the size of active memory in bytes. µ′s[0] ≡ 32µi 0x5a gas 0 1 get the amount of available gas, including the corresponding reduction for the cost of this instruction. µ′s[0] ≡ µg 0x5b jumpdest 0 0 mark a valid destination for jumps. this operation has no effect on machine state during execution. 5f, 60s & 70s: push operations value mnemonic δ α description 0x5f push0 0 1 place 0 on the stack. µ′s[0] ≡ 0 0x60 push1 0 1 place 1 byte item on stack. µ′s[0] ≡ c(µpc + 1) where c(x) ≡ { ib[x] if x < ‖ib‖ 0 otherwise the bytes are read in line from the program code’s bytes array. the function c ensures the bytes default to zero if they extend past the limits. the byte is right-aligned (takes the lowest significant place in big endian). 0x61 push2 0 1 place 2-byte item on stack. µ′s[0] ≡ c ( (µpc + 1) . . . (µpc + 2) ) with c(x) ≡ (c(x0), ...,c(x‖x‖−1)) with c as defined as above. the bytes are right-aligned (takes the lowest significant place in big endian). ... ... ... ... ... 0x7f push32 0 1 place 32-byte (full word) item on stack. µ′s[0] ≡ c ( (µpc + 1) . . . (µpc + 32) ) where c is defined as above. the bytes are right-aligned (takes the lowest significant place in big endian). 80s: duplication operations value mnemonic δ α description 0x80 dup1 1 2 duplicate 1st stack item. µ′s[0] ≡ µs[0] 0x81 dup2 2 3 duplicate 2nd stack item. µ′s[0] ≡ µs[1] ... ... ... ... ... 0x8f dup16 16 17 duplicate 16th stack item. µ′s[0] ≡ µs[15] 90s: exchange operations value mnemonic δ α description 0x90 swap1 2 2 exchange 1st and 2nd stack items. µ′s[0] ≡ µs[1] µ′s[1] ≡ µs[0] 0x91 swap2 3 3 exchange 1st and 3rd stack items. µ′s[0] ≡ µs[2] µ′s[2] ≡ µs[0] ... ... ... ... ... 0x9f swap16 17 17 exchange 1st and 17th stack items. µ′s[0] ≡ µs[16] µ′s[16] ≡ µs[0] ethereum: a secure decentralised generalised transaction ledger shanghai version 37 a0s: logging operations for all logging operations, the state change is to append an additional log entry on to the substate’s log series: a′l ≡ al · (ia, t,µm[µs[0] . . . (µs[0] + µs[1] − 1)]) and to update the memory consumption counter: µ′i ≡ m(µi,µs[0],µs[1]) the entry’s topic series, t, differs accordingly: value mnemonic δ α description 0xa0 log0 2 0 append log record with no topics. t ≡ () 0xa1 log1 3 0 append log record with one topic. t ≡ (µs[2]) ... ... ... ... ... 0xa4 log4 6 0 append log record with four topics. t ≡ (µs[2],µs[3],µs[4],µs[5]) ethereum: a secure decentralised generalised transaction ledger shanghai version 38 f0s: system operations value mnemonic δ α description 0xf0 create 3 1 create a new account with associated code. i ≡ µm[µs[1] . . . (µs[1] + µs[2] − 1)] ζ ≡ ∅ (σ′,g′,a′,z, o) ≡   λ(σ∗,a,ia,io,l(µg),ip,µs[0], i,ie + 1,ζ,iw) if µs[0] 6 σ[ia]b ∧ ie < 1024 ∧‖i‖6 49152( σ,l(µg),a, 0, () ) otherwise σ∗ ≡ σ except σ∗[ia]n = σ[ia]n + 1 µ′g ≡ µg −l(µg) + g ′ µ′s[0] ≡ x where x = 0 if z = 0, i.e., the contract creation process failed, or ‖i‖ > 49152 (initcode is longer than 49152 bytes), or ie = 1024 (the maximum call depth limit is reached), or µs[0] > σ[ia]b (balance of the caller is too low to fulfil the value transfer); and otherwise x = addr(ia,σ[ia]n,ζ, i), the address of the newly created account (95). µ′i ≡ m(µi,µs[1],µs[2]) µ′o ≡ { () if z = 1 o otherwise thus the operand order is: value, input offset, input size. 0xf1 call 7 1 message-call into an account. i ≡ µm[µs[3] . . . (µs[3] + µs[4] − 1)] (σ′,g′,a′,x, o) ≡   θ(σ,a∗,ia,io, t, t,ccallgas(σ,µ,a), ip,µs[2],µs[2], i,ie + 1,iw) if µs[2] 6 σ[ia]b ∧ ie < 1024 (σ,ccallgas(σ,µ,a),a ∗, 0, ()) otherwise n ≡ min({µs[6],‖o‖}) µ′m[µs[5] . . . (µs[5] + n− 1)] = o[0 . . . (n− 1)] µ′o = o µ′g ≡ µg −ccallgas(σ,µ,a) + g ′ µ′s[0] ≡ x a∗ ≡ a except a∗a ≡ aa ∪{t} t ≡ µs[1] mod 2 160 µ′i ≡ m(m(µi,µs[3],µs[4]),µs[5],µs[6]) where x = 0 if the code execution for this operation failed, or if µs[2] > σ[ia]b (not enough funds) or ie = 1024 (call depth limit reached); x = 1 otherwise. thus the operand order is: gas, to, value, in offset, in size, out offset, out size. ccall(σ,µ,a) ≡ cgascap(σ,µ,a) + cextra(σ,µ,a) ccallgas(σ,µ,a) ≡ { cgascap(σ,µ,a) + gcallstipend if µs[2] 6= 0 cgascap(σ,µ,a) otherwise cgascap(σ,µ,a) ≡ { min{l(µg −cextra(σ,µ,a)),µs[0]} if µg ≥ cextra(σ,µ,a) µs[0] otherwise cextra(σ,µ,a) ≡ caaccess(t,a) + cxfer(µ) + cnew(σ,µ) cxfer(µ) ≡ { gcallvalue if µs[2] 6= 0 0 otherwise cnew(σ,µ) ≡ { gnewaccount if dead(σ, t) ∧µs[2] 6= 0 0 otherwise 0xf2 callcode 7 1 message-call into this account with an alternative account’s code. exactly equivalent to call except: (σ′,g′,a′,x, o) ≡   θ(σ,a∗,ia,io,ia, t,ccallgas(σ,µ,a), ip,µs[2],µs[2], i,ie + 1,iw) if µs[2] 6 σ[ia]b ∧ ie < 1024 (σ,ccallgas(σ,µ,a),a ∗, 0, ()) otherwise note the change in the fourth parameter to the call θ from the 2nd stack value µs[1] (as in call) to the present address ia. this means that the recipient is in fact the same account as at present, simply that the code is overwritten. 0xf3 return 2 0 halt execution returning output data. hreturn(µ) ≡ µm[µs[0] . . . (µs[0] + µs[1] − 1)] this has the effect of halting the execution at this point with output defined. see section 9. µ′i ≡ m(µi,µs[0],µs[1]) ethereum: a secure decentralised generalised transaction ledger shanghai version 39 0xf4 delegatecall 6 1 message-call into this account with an alternative account’s code, but persisting the current values for sender and value. compared with call, delegatecall takes one fewer arguments. the omitted argument is µs[2]. as a result, µs[3], µs[4], µs[5] and µs[6] in the definition of call should respectively be replaced with µs[2], µs[3], µs[4] and µs[5]. otherwise it is equivalent to call except: (σ′,g′,a′,x, o) ≡   θ(σ,a∗,is,io,ia, t,ccallgas(σ,µ,a), ip, 0,iv, i,ie + 1,iw) if ie < 1024 (σ,ccallgas(σ,µ,a),a ∗, 0, ()) otherwise note the changes (in addition to that of the fourth parameter) to the second and ninth parameters to the call θ. this means that the recipient is in fact the same account as at present, simply that the code is overwritten and the context is almost entirely identical. 0xf5 create2 4 1 create a new account with associated code. exactly equivalent to create except: the salt ζ ≡ µs[3]. 0xfa staticcall 6 1 static message-call into an account. exactly equivalent to call except: the argument µs[2] is replaced with 0. the deeper argument µs[3], µs[4], µs[5] and µs[6] are respectively replaced with µs[2], µs[3], µs[4] and µs[5]. the last argument of θ is ⊥. 0xfd revert 2 0 halt execution reverting state changes but returning data and remaining gas. hreturn(µ) ≡ µm[µs[0] . . . (µs[0] + µs[1] − 1)] the effect of this operation is described in (152). for the gas calculation, we use the memory expansion function, µ′i ≡ m(µi,µs[0],µs[1]) 0xfe invalid ∅ ∅ designated invalid instruction. 0xff selfdestruct 1 0 halt execution and register account for later deletion. readers should be warned, since the shanghai update, this opcode has been marked as deprecated. its behavior might be subject to breaking change in an upcoming update. a′s ≡ as ∪{ia} a′a ≡ aa ∪{r} σ′[r] ≡   ∅ if σ[r] = ∅ ∧ σ[ia]b = 0 (σ[r]n,σ[r]b + σ[ia]b,σ[r]s,σ[r]c) if r 6= ia (σ[r]n, 0,σ[r]s,σ[r]c) otherwise where r = µs[0] mod 2 160 σ′[ia]b = 0 cselfdestruct(σ,µ) ≡ gselfdestruct + { 0 if r ∈ aa gcoldaccountaccess otherwise + { gnewaccount if dead(σ,r) ∧σ[ia]b 6= 0 0 otherwise appendix i. genesis block the genesis block is 15 items, and is specified thus: (333) (( 0256,kec ( rlp ( () )) , 0160,stateroot, 0, 0, 02048, 2 34 , 0, 0, 3141592, time, 0, 0256,kec ( (42) )) , (), () ) where 0256 refers to the parent hash, a 256-bit hash which is all zeroes; 0160 refers to the beneficiary address, a 160-bit hash which is all zeroes; 02048 refers to the log bloom, 2048-bit of all zeros; 2 34 refers to the difficulty; the transaction trie root, receipt trie root, gas used, block number and extradata are both 0, being equivalent to the empty byte array. the sequences of both ommers and transactions are empty and represented by (). kec ( (42) ) refers to the keccak-256 hash of a byte array of length one whose first and only byte is of value 42, used for the nonce. kec ( rlp ( () )) value refers to the hash of the ommer list in rlp, both empty lists. the proof-of-concept series include a development premine, making the state root hash some value stateroot. also time will be set to the initial timestamp of the genesis block. the latest documentation should be consulted for those values. ethereum: a secure decentralised generalised transaction ledger shanghai version 40 appendix j. ethash j.1. deprecation. this section is kept for historical purposes, but the ethash algorithm is no longer used for consensus as of the paris hard fork. j.2. definitions. we employ the following definitions: name value description jwordbytes 4 bytes in word. jdatasetinit 2 30 bytes in dataset at genesis. jdatasetgrowth 2 23 dataset growth per epoch. jcacheinit 2 24 bytes in cache at genesis. jcachegrowth 2 17 cache growth per epoch. jepoch 30000 blocks per epoch. jmixbytes 128 mix length in bytes. jhashbytes 64 hash length in bytes. jparents 256 number of parents of each dataset element. jcacherounds 3 number of rounds in cache production. jaccesses 64 number of accesses in hashimoto loop. j.3. size of dataset and cache. the size for ethash’s cache c ∈ b and dataset d ∈ b depend on the epoch, which in turn depends on the block number. (334) eepoch(hi) = ⌊ hi jepoch ⌋ the size of the dataset growth by jdatasetgrowth bytes, and the size of the cache by jcachegrowth bytes, every epoch. in order to avoid regularity leading to cyclic behavior, the size must be a prime number. therefore the size is reduced by a multiple of jmixbytes, for the dataset, and jhashbytes for the cache. let dsize = ‖d‖ be the size of the dataset. which is calculated using (335) dsize = eprime(jdatasetinit + jdatasetgrowth ·eepoch −jmixbytes,jmixbytes) the size of the cache, csize, is calculated using (336) csize = eprime(jcacheinit + jcachegrowth ·eepoch −jhashbytes,jhashbytes) (337) eprime(x,y) = { x if x/y ∈ n eprime(x− 2 ·y,y) otherwise j.4. dataset generation. in order to generate the dataset we need the cache c, which is an array of bytes. it depends on the cache size csize and the seed hash s ∈ b32. j.4.1. seed hash. the seed hash is different for every epoch. for the first epoch it is the keccak-256 hash of a series of 32 bytes of zeros. for every other epoch it is always the keccak-256 hash of the previous seed hash: (338) s = cseedhash(hi) (339) cseedhash(hi) = { 032 if eepoch(hi) = 0 kec(cseedhash(hi −jepoch)) otherwise with 032 being 32 bytes of zeros. j.4.2. cache. the cache production process involves using the seed hash to first sequentially filling up csize bytes of memory, then performing jcacherounds passes of the randmemohash algorithm created by lerner [2014]. the initial cache c′, being an array of arrays of single bytes, will be constructed as follows. we define the array ci, consisting of 64 single bytes, as the ith element of the initial cache: (340) ci = { kec512(s) if i = 0 kec512(ci−1) otherwise therefore c′ can be defined as (341) c ′ [i] = ci ∀ i < n (342) n = ⌊ csize jhashbytes ⌋ the cache is calculated by performing jcacherounds rounds of the randmemohash algorithm to the initial cache c ′: (343) c = ecacherounds(c ′ ,jcacherounds) ethereum: a secure decentralised generalised transaction ledger shanghai version 41 (344) ecacherounds(x,y) =   x if y = 0 ermh(x) if y = 1 ecacherounds(ermh(x),y − 1) otherwise where a single round modifies each subset of the cache as follows: (345) ermh(x) = ( ermh(x, 0),ermh(x, 1), ...,ermh(x,n− 1) ) (346) ermh(x, i) = kec512(x ′ [(i− 1 + n) mod n] ⊕ x′[x′[i][0] mod n]) with x ′ = x except x ′ [j] = ermh(x,j) ∀ j < i j.4.3. full dataset calculation. essentially, we combine data from jparents pseudorandomly selected cache nodes, and hash that to compute the dataset. the entire dataset is then generated by a number of items, each jhashbytes bytes in size: (347) d[i] = edatasetitem(c, i) ∀ i < ⌊ dsize jhashbytes ⌋ in order to calculate the single item we use an algorithm inspired by the fnv hash (glenn fowler [1991]) in some cases as a non-associative substitute for xor. (348) efnv(x, y) = (x · (0x01000193 ⊕ y)) mod 232 the single item of the dataset can now be calculated as: (349) edatasetitem(c, i) = eparents(c, i,−1,∅) (350) eparents(c, i,p, m) = { eparents(c, i,p + 1,emix(m, c, i,p + 1)) if p < jparents − 2 emix(m, c, i,p + 1) otherwise (351) emix(m, c, i,p) = { kec512(c[i mod csize] ⊕ i) if p = 0 efnv ( m, c[efnv(i⊕p, m[p mod bjhashbytes/jwordbytesc]) mod csize] ) otherwise j.5. proof-of-work function. essentially, we maintain a “mix” jmixbytes bytes wide, and repeatedly sequentially fetch jmixbytes bytes from the full dataset and use the efnv function to combine it with the mix. jmixbytes bytes of sequential access are used so that each round of the algorithm always fetches a full page from ram, minimizing translation lookaside buffer misses which asics would theoretically be able to avoid. if the output of this algorithm is below the desired target, then the nonce is valid. note that the extra application of kec at the end ensures that there exists an intermediate nonce which can be provided to prove that at least a small amount of work was done; this quick outer pow verification can be used for anti-ddos purposes. it also serves to provide statistical assurance that the result is an unbiased, 256-bit number. the pow-function returns an array with the compressed mix as its first item and the keccak-256 hash of the concatenation of the compressed mix with the seed hash as the second item: (352) pow(hn,hn, d) = {mc(kec(rlp(lh(hn))),hn, d),kec(sh(kec(rlp(lh(hn))),hn) + mc(kec(rlp(lh(hn))),hn, d))} with hn being the hash of the header without the nonce. the compressed mix mc is obtained as follows: (353) mc(h, n, d) = ecompress(eaccesses(d, nmix∑ i=0 sh(h, n), sh(h, n),−1),−4) the seed hash being: (354) sh(h, n) = kec512(h + erevert(n)) erevert(n) returns the reverted bytes sequence of the nonce n: (355) erevert(n)[i] = n[‖n‖− i] we note that the “+”-operator between two byte sequences results in the concatenation of both sequences. the dataset d is obtained as described in section j.4.3. the number of replicated sequences in the mix is: (356) nmix = ⌊ jmixbytes jhashbytes ⌋ in order to add random dataset nodes to the mix, the eaccesses function is used: (357) eaccesses(d, m, s, i) = { emixdataset(d, m, s, i) if i = jaccesses − 2 eaccesses(emixdataset(d, m, s, i), s, i + 1) otherwise (358) emixdataset(d, m, s, i) = efnv(m,enewdata(d, m, s, i)) ethereum: a secure decentralised generalised transaction ledger shanghai version 42 enewdata returns an array with nmix elements: (359) enewdata(d, m, s, i)[j] = d[efnv(i⊕ s[0], m[i mod ⌊ jmixbytes jwordbytes ⌋ ]) mod ⌊ dsize/jhashbytes nmix ⌋ ·nmix + j] ∀ j < nmix the mix is compressed as follows: (360) ecompress(m, i) = { m if i > ‖m‖− 8 ecompress(efnv(efnv(efnv(m[i + 4], m[i + 5]), m[i + 6]), m[i + 7]), i + 8) otherwise appendix k. anomalies on the main network k.1. deletion of an account despite out-of-gas. at block 2675119, in the transaction 0xcf416c536ec1a19ed1fb89e 4ec7ffb3cf73aa413b3aa9b77d60e4fd81a4296ba, an account at address 0x03 was called and an out-of-gas occurred during the call. against the equation (209), this added 0x03 in the set of touched addresses, and this transaction turned σ[0x03] into ∅. appendix l. list of mathematical symbols symbol latex command description∨ \bigvee this is the least upper bound, supremum, or join of all elements operated on. thus it is the greatest element of such elements (davey and priestley [2002]). 1. introduction 1.1. driving factors 1.2. previous work 2. the blockchain paradigm 2.1. value 2.2. which history? 3. conventions 4. blocks, state and transactions 4.1. world state 4.2. the transaction 4.3. the withdrawal 4.4. the block 5. gas and payment 6. transaction execution 6.1. substate 6.2. execution 7. contract creation 7.1. subtleties 8. message call 9. execution model 9.1. basics 9.2. fees overview 9.3. execution environment 9.4. execution overview 9.5. the execution cycle 10. transition to proof of stake 10.1. post-paris updates 11. blocktree to blockchain 12. block finalisation 12.1. executing withdrawals 12.2. transaction validation 12.3. state validation 13. implementing contracts 13.1. data feeds 13.2. random numbers 14. future directions 15. conclusion 16. acknowledgements 17. availability references appendix a. terminology appendix b. recursive length prefix appendix c. hex-prefix encoding appendix d. modified merkle patricia tree d.1. trie database appendix e. precompiled contracts e.1. zksnark related precompiled contracts e.2. blake2 precompiled contract appendix f. signing transactions appendix g. fee schedule appendix h. virtual machine specification h.1. gas cost h.2. instruction set appendix i. genesis block appendix j. ethash j.1. deprecation j.2. definitions j.3. size of dataset and cache j.4. dataset generation j.5. proof-of-work function appendix k. anomalies on the main network k.1. deletion of an account despite out-of-gas appendix l. list of mathematical symbols