The following were results obtained by mixing Resveratrol (that was in a japanese knot weed concentrate) with DMSO and applying for 10 days, about twice a day. The scar on top of head was 10 years old from scraping scalp off on an overhang ledge (in 2006). It had gotten worse than the 2013 picture and then I bumped it recently and it was swelling up with a scab that was slow in healing (see picture), so I applied DMSO with resveratrol. The Feb pic was actually after 5 treatments and I stopped a few days later. I did not think much else about it, but you can see from the march picture that it finished healing on its on. It appears the DMSO/Resv kills the cells and stains them then it takes a month or more to clear out the dead/stained material.
The Mole/age spot/whatever (keratosis since unlike age spots it was a little bumpy?) on the side of the head started about 6 years ago and was beginning to look a little frightening, getting raised above the skin 0.5 mm. I have a close up of it that shows how remarkable this is.
Since I wrote the above I've tried it on several other moles and spots. It does not always work, at least not after about 15 days of twice-daily treatments. But I had some serious sun damage on my shin from old exposure and it had a remarkable effect there too. I have picture of that plus the moles provably better (they might be better). The spots on my shoulders have returned a little after a year, but they are still a lot better.
Tuesday, October 4, 2016
Thursday, September 22, 2016
Cryptocoins equally to all people w/o 3rd party OR transaction fee feedback to create constant value coin
Maybe there is a way to issue a fixed quantity of coin to all people on Earth without a 3rd party.
Your biomeasures are different kinds of "hashes" of your genes (and environment and chance). The following might work because single genes affect multiple systems. Given the right set of biomeasures it may not be feasible to generate a valid survivable human DNA sequence. One biomeasure constrains DNA one way, and another in another way, and so on. But given the biomeasures and DNA sequence the blockchain might prove a given pairing is valid. People would use the set of biomeasures and their DNA to apply to the blockchain for coins and a private key. DNA and private key would generate wallet addresses.
The key is that each gene usually affects multiple biometric measures, maybe in the same way a prime can be used to generate many different public keys when combined with other primes. Or maybe I should view the biometric measures as a hash of the genes. Either way, there seems to be a 1-way function that can be exploited. You can get biometrics from genes, but maybe not valid genes from biometrics.
Genes causing the expression of biometrics (genotype creates phenotype) is such a messy business (a huge and messy kind of hashing, not subject to strict mathematics and influenced by environment and randomness), traditional cryptography might not be usable. At first it might require a world class neural net to get started, then the blockchain would have to take over as the neural net. The neural net would take all available DNA and biometric data and find all patterns backwards and forwards (genes -> biometrics, biometrics -> genes) that it can. It would attempt to predict viable DNA from biometrics and vice versa. The vice versa (determining biometrics from genes) is relatively easy, but we are in its infancy. A lot of medical research is doing this because having a disease is a biometric result of the genes. But getting DNA from biometrics could be made very difficult if the right biometrics are chosen. A neural net could predict viable biometrics from DNA, but my thesis is that it could be really difficult to create viable DNA from a correctly chosen set of measured biometrics. The neural net's job is to discover the best biometrics to use (the ones it can't crack), and to constantly try to crack it. Successful cracks are rewarded. Along the way it is discovering what genes do as the preliminary step to cracking (it has to get its list of "primes"?).
Since population growth I think is around 2% and slowing, the inflation problem should be small, and even a benefit as I stated before, in contradiction to the usual cryptocoin beliefs concerning fixed-quantity coins.
It seems I am requiring people to apply for their coins using their biometric and DNA data before others get their DNA and generate viable biometrics.
BTW, a 3rd party is always present if the code can be changed at any time after launch. Developers being guided by users is the same as government being guided by voters. Lobbies like the rich or bankers (PoS and miners) that subvert the users' voting process is the same system we have for the dollar. Observational evidence for this viiwpoint: we seek ethics in the developers in the same way we seek ethics in government leaders.Your biomeasures are different kinds of "hashes" of your genes (and environment and chance). The following might work because single genes affect multiple systems. Given the right set of biomeasures it may not be feasible to generate a valid survivable human DNA sequence. One biomeasure constrains DNA one way, and another in another way, and so on. But given the biomeasures and DNA sequence the blockchain might prove a given pairing is valid. People would use the set of biomeasures and their DNA to apply to the blockchain for coins and a private key. DNA and private key would generate wallet addresses.
The key is that each gene usually affects multiple biometric measures, maybe in the same way a prime can be used to generate many different public keys when combined with other primes. Or maybe I should view the biometric measures as a hash of the genes. Either way, there seems to be a 1-way function that can be exploited. You can get biometrics from genes, but maybe not valid genes from biometrics.
Genes causing the expression of biometrics (genotype creates phenotype) is such a messy business (a huge and messy kind of hashing, not subject to strict mathematics and influenced by environment and randomness), traditional cryptography might not be usable. At first it might require a world class neural net to get started, then the blockchain would have to take over as the neural net. The neural net would take all available DNA and biometric data and find all patterns backwards and forwards (genes -> biometrics, biometrics -> genes) that it can. It would attempt to predict viable DNA from biometrics and vice versa. The vice versa (determining biometrics from genes) is relatively easy, but we are in its infancy. A lot of medical research is doing this because having a disease is a biometric result of the genes. But getting DNA from biometrics could be made very difficult if the right biometrics are chosen. A neural net could predict viable biometrics from DNA, but my thesis is that it could be really difficult to create viable DNA from a correctly chosen set of measured biometrics. The neural net's job is to discover the best biometrics to use (the ones it can't crack), and to constantly try to crack it. Successful cracks are rewarded. Along the way it is discovering what genes do as the preliminary step to cracking (it has to get its list of "primes"?).
Since population growth I think is around 2% and slowing, the inflation problem should be small, and even a benefit as I stated before, in contradiction to the usual cryptocoin beliefs concerning fixed-quantity coins.
It seems I am requiring people to apply for their coins using their biometric and DNA data before others get their DNA and generate viable biometrics.
There is another way to achieve a constant-value coin that is a lot less difficult than using DNA, but does not retain the virtue of blocking machines out of human economics. **Let the market-determined transaction fees per coin determine the coin release-rate.** If the fee rises there is a shortage of nodes compared to daily coin transaction volume. Additional fees per byte and a base fee per transaction would be needed, but not used to determine the coin release rate. This uses the velocity of money theory. So the developers are not allowed (and not required) to decide the final quantity or release schedule of the coin. The market does. A PID controller would take the transaction fee per coin as the input and output the coins per block. If the fees drop too much, it indicates the coin is not being used much and coins per block can go to zero, keeping coin quantity constant. Miners would stop mining and nodes would live off the base fee for transactions. Another controller would take the number of nodes per transaction as the input and drop the base fee and/or per byte fee if the ratio of nodes to transactions got unnecessarily high, which keeps the coin competitive and lean without oversight. The more feedback controllers used intelligently, the more intelligent the coin (and anything else) is.
I am not saying the above is perfectly correct or complete. I wanted to show that some idea like it could create the cryptocurrency holy grail: a constant value coin not based on perception, opinion, miners, or developers.
Intelligent direction (i.e. controller feedback) of permission (i.e. legal tender, aka currency) to use available resources is the basis of all intelligence. Be it glucose and molecules in the brain, energy and matter in economics, or CPU time (kinetic joules=expenses) and RAM/HDD space (potential joules=initial investment) in computing, the intelligent direction of the currency directs the energy and matter for personal profit (growth based on more and more energy and matter coming under control of the movement of the currency). Democracy uses the feedback of votes to guide the taxes which directs the energy and matter in government which a controller on the economics which gives voters what they want. The most intelligence cryptocoin will be a growing, spreading, changing A.I. of feedback controllers (smart contracts directing the coin) that enables the market place that falls under its control to be the most profitable and growing so that the cryptocoin itself can be profitable and grow by riding (lightly) on its back so that it is a symbiotic relation instead of viral/cancerous. The end goal is congeal the matter on Earth into a more ordered form, releasing entropy to the universe. We are doing this by shifting from organic bonds to metal and metalloid bonds, removing oxygen from metals, silicon, and carbon so that we have greater control through lower entropy per kg of our economic machine. Earth's unusual because of the order injected by the Moon, and why we look for life on Titan and Io (geological disturbances are cyclic forces that inject order into thermodynamically-stable systems).
The market itself is just a bunch of feedback going on between agents, under the rules of some governing coin (i.e. legal tender). So ideally, the feedback systems would probably be nested and complicated from bottom to top so that the distinction between government and market is not clear, while the coin would be very clear. Separate "organs" of law (code) could easily have their own internal coins, but still be based on a system wide coin. Maybe the highest level coin describes the boundaries and definition of an entity. The highest I know of is energy (Gibbs free energy). Maybe there is some sort of negative entropy that could be higher. But a single coin and system without distinguishable "organs" should be the most efficient, like a highly compressed algorithm.
But for current work on cryptocurrencies, it seems 1 to 5 feedback measures should be the limit.
There is currently no feedback from the market place (other than the difficulty) to tell cryptocoins how the coins are to be issued in order to best benefit the market. The arbitrary nature of coin quantity, release schedule, and fees needs to be changed and connected to the coin's usage and computational power.
=====
Let transaction fee per coin control coins per block issued and never let difficulty fall. Problem solved? A base fee per transaction and fee per byte would also be needed. A standard PID controller on the transaction "error signal" would be used. Difficulty can easily get too high, but there is no incentive for attacks to make it go high because they can't profit on downturns. Large miners can't profit from random difficulty swings or manipulate it for profit. If difficulty is too high, miners will get out if fees are not high enough. But surviving this demonstrates the system is not a Ponzi scheme that will end when mining ends. A decrease in network hash rate might adjust the set point that the transaction fee error signal needs. With the right feedback (checks and balances) developers would not be required (or allowed) to choose any aspects of coin issuance (not total quantity, schedule, coins/block, difficulty, or fees). The market should be able to dictate everything without anyone needing to explicitly vote except by their marketplace choices (miners getting in or out, and transaction fees). If the market for the coin starts to dry up (it's fees were too high to sustain miners) then it merely shows a more efficient coin is taking its place, and it should dry up. But the quantity of the at the point is constant.
Friday, September 9, 2016
Sunday, August 28, 2016
Xanthohumol in beer hops is a potent nilotinib-like tyrosine kinase inhibitor
Saturday, August 27, 2016
GPUs vs CPUs in Zcash
An 8 GB GPU could run 2.5 more threads at 8x more bandwidth by simply porting the CPU code (no special parallelizing needed). The problem with the Equihash paper is that it reverenced a 2011 GPU and did not point out that modern GPUs have a lot more on-board RAM. So 2.5 x 8 = 20x is an upper limit. But the cores are operating at 1/3 the clock speed of CPUs and my experiments in acquiring blocks on the testnest indicate core-caching and/or clock speed on the CPU matters a lot. Either way, it indicates less than 2.5 x 8, maybe 8x benefit as a minimum. The important point is that this minimum is double the Equihash paper and it does not require any special programming that was required in the 4x claim of the Equihash paper. The paper referenced a 2011 CPU for the comparison, so I did not think there was a problem in looking at an old GPU as both have advanced. So the problem (if you wanted CPUs instead of GPUs) is that Zcash has chosen parameters that are good for 2011 but not for 2016. I am not being critical as I did not realize the implications myself until now. Even without the GUI, I could not get 4 threads to run good on 4 GB, and 6GB seemed to be even worse. So 8 GB is the demand. Since 8 GB is the demand, 750 MB/thread is not good. 1.2 GB should have been the requirement in order to allow ubuntu and to hold back GPUs.
=====
update to the above:
==========
The Equihash paper was from 2016. The GPU vs CPU data was from 2011. I wanted nothing more than CPUs to win, but an 8 GB GPU should be 10x better than a CPU at launch if they are no better than the stock miner. The Equihash paper assumed the cryptocurrency developers would choose a RAM requirement that is higher than on-board GPU RAM. But with new GPUs, a GPU coder can copy the stock miner and run it on 10 cores to get 2.5x more threads than a 4 core CPU at 20x the bandwidth (a $400 GPU). It's not 20 x 2.5 = 50x faster than CPUs only because the GPU cores are so slow. The 4x statement in the Equihash has nothing to do with this: by assuming the coin's RAM requirement would be larger than the GPU RAM, they assumed advanced parallel programming would be needed to make use of the GPU's many cores. That is not the case. Zcash was not able to force larger RAM, so the Equihash paper is not relevant as far as GPUs are concerned. They might could make the RAM about 1200 MB per core if they go to 5 minute intervals. This would reduce the GPU advantage to 7.5 by my above math.
But we have not actually seen any $400 GPU results faster than a $200 desktop.
=====
update to the above:
==========
The Equihash paper was from 2016. The GPU vs CPU data was from 2011. I wanted nothing more than CPUs to win, but an 8 GB GPU should be 10x better than a CPU at launch if they are no better than the stock miner. The Equihash paper assumed the cryptocurrency developers would choose a RAM requirement that is higher than on-board GPU RAM. But with new GPUs, a GPU coder can copy the stock miner and run it on 10 cores to get 2.5x more threads than a 4 core CPU at 20x the bandwidth (a $400 GPU). It's not 20 x 2.5 = 50x faster than CPUs only because the GPU cores are so slow. The 4x statement in the Equihash has nothing to do with this: by assuming the coin's RAM requirement would be larger than the GPU RAM, they assumed advanced parallel programming would be needed to make use of the GPU's many cores. That is not the case. Zcash was not able to force larger RAM, so the Equihash paper is not relevant as far as GPUs are concerned. They might could make the RAM about 1200 MB per core if they go to 5 minute intervals. This would reduce the GPU advantage to 7.5 by my above math.
But we have not actually seen any $400 GPU results faster than a $200 desktop.
Thursday, August 11, 2016
Zcash and energy-expense in cryptocoins may not be a bad thing
I mentioned computers are using 4x more
electricity when running Zcash. It may make GPU's less capable of
competing. They are not able to access the external RAM directly, so
they are less efficient, having to compute more per hash. The 4x speed
of parallel code for GPUs of the future will come with at least 2x more
energy cost.
From measurements on the testnet and my electricity use, if there are 50,000 PCs on the network, it will cost $1 per coin in electricity above 24 hour normal-use PC costs if you have a 22 nm process CPU that is meant to be efficient ( less than 3 GHz).
Although the high energy use is against what most people consider a "good" coin, it might be an inherent necessity if POW is inherently needed. The high energy use is key to making mining widely distributed. If the only thing determining the quantity of coin obtainable is the amount of energy purchased, then most people have equal access. Electricity rates can vary a lot compared to oil (e.g. islands & remote villages), but that is a small portion of the world's population. If a government subsidizes cheap electricity by investment or allowing more pollution, then the populace of that region have paid the price that local miners gain. If they optimize code and have cheap electricity, they might get 5x more coin per dollar expense compared to small miners.
If Zcash spreads to the populace who do not have to buy equipment and do not even notice they have higher electrical costs, mining may not be feasible. This is the stated goal. This means a general-purpose CPU needs to be biased for. This means more electricity to stretch its general-purpose skills. Sorts seem very specific, but they bias everything towards a more general purpose Turing machine. The entire basis of a Turing machine is reading and writing, and sorts need it in ways that are hard optimize in a way that reduces the need to read and write.
The RAM of devices is not generally being wasted like CPU time, so it might be better to be be CPU-centric. But part of the path to the general-purpose CPU is high RAM in order to block out non-general purpose GPUs and ASICs.
So it's a coin that promotes generalized computing devices in everyone's hands without taking away too much RAM, rather than wasting money on specific equipment for specific people (miners). This is a 2nd reason a higher electrical expense is not a bad idea: CPU time is being wasted more than RAM space. And note that RAM is not electricity-free. There is a very large initial electrical expense in creating the RAM, as indicated by it's price. This indicates equal use of CPU and RAM may be better as one is an on-going time-based expense of Joules and the other is a one-time capital "space-based" expense of Joules. CPUs require a Joules per bit state change in time, and RAM requires a Joules construction cost per bit storage space in cm^3. Of course RAM has state-change energy cost and CPU has construction cost, but those energy costs are smaller.
All economic schools have said following a basket of commodities is the best currency. Those favoring gold do so only because it is the best single commodity that has the right properties. It also one of the most wasteful ways to use kinetic energy, which is key to its perceived value. A basket would require measurements and "paper" (bits on a computer). The cost of energy (like electricity) is the largest underlying factor in the cost of producing commodities. So currencies based on Joules have been proposed as ideal. Zcash is a Joules-based currency. The Joules-as-value, both kinetic and potential, has a deep connection with computation and life. (see Note at bottom).
There is a 4th "benefit" to a high electrical cost per coin, although all these points are connected. It should not sell for less than the cost to produce it, unless someone has given up on the coin and will accept a loss.
Zcash's goal is to be "democratic" in mining. The result is an ideal cryptocoin. POW should not be "might is right", but "distributed might is right". Otherwise, the ability of miners to lobby the governing coders becomes the wrong kind of might.
This is not to say an energy-intensive coin is best for society. A coin that is given based on how much a person helps society (such as Solar Coin) would be best. But that involves agreement on definition of what is "best" (are solar cells really the best use of your money to be subsidized by giving you a coin?) and then measuring it before the cryptography part can even begin. It is a type of fiat requiring a middle man (or at least a group of oracles that are doing an agreed upon measurement, governed by smart contracts that define the rules for the distribution use of a specific coin). The whole reason fiat replaced gold is because governments are able to print it and distribute it evenly based on achieving goals that are ostensibly best for that society. Coins distributed based on POW that is not connected with the betterment of society are not best unless the government is not acting in the best interest of people and/or anarchy (e.g., hyperinflation) is near.
Note: To be more correct, it is the Joules as measured by Gibb's Free Energy that is essential to life. Schrodinger even updated his "What Is Life?" book to point out that when he said "negative entropy" as being the key to life, he really meant free energy. Gibb's F.E. = U -TS where U is internal energy and TS is temp x entropy. In terms of Zcash, U=RAM+CPU capital one-shot "energy" expense and TS=CPU operating on-going energy expense. The first is energy stored in spacial RAM and CPU structures, and the second is energy spent in time. CPU computations come at an energy cost of TS/eff where eff is the efficiency of the device. This does not include algorithm efficiency. Per N bits that change state irreversibly, the number of Joules expended is T x S / eff (see Landauer's limit relating bits to physical entropy) where S=kb x ln(2) x N. For a given algorithm.
From measurements on the testnet and my electricity use, if there are 50,000 PCs on the network, it will cost $1 per coin in electricity above 24 hour normal-use PC costs if you have a 22 nm process CPU that is meant to be efficient ( less than 3 GHz).
Although the high energy use is against what most people consider a "good" coin, it might be an inherent necessity if POW is inherently needed. The high energy use is key to making mining widely distributed. If the only thing determining the quantity of coin obtainable is the amount of energy purchased, then most people have equal access. Electricity rates can vary a lot compared to oil (e.g. islands & remote villages), but that is a small portion of the world's population. If a government subsidizes cheap electricity by investment or allowing more pollution, then the populace of that region have paid the price that local miners gain. If they optimize code and have cheap electricity, they might get 5x more coin per dollar expense compared to small miners.
If Zcash spreads to the populace who do not have to buy equipment and do not even notice they have higher electrical costs, mining may not be feasible. This is the stated goal. This means a general-purpose CPU needs to be biased for. This means more electricity to stretch its general-purpose skills. Sorts seem very specific, but they bias everything towards a more general purpose Turing machine. The entire basis of a Turing machine is reading and writing, and sorts need it in ways that are hard optimize in a way that reduces the need to read and write.
The RAM of devices is not generally being wasted like CPU time, so it might be better to be be CPU-centric. But part of the path to the general-purpose CPU is high RAM in order to block out non-general purpose GPUs and ASICs.
So it's a coin that promotes generalized computing devices in everyone's hands without taking away too much RAM, rather than wasting money on specific equipment for specific people (miners). This is a 2nd reason a higher electrical expense is not a bad idea: CPU time is being wasted more than RAM space. And note that RAM is not electricity-free. There is a very large initial electrical expense in creating the RAM, as indicated by it's price. This indicates equal use of CPU and RAM may be better as one is an on-going time-based expense of Joules and the other is a one-time capital "space-based" expense of Joules. CPUs require a Joules per bit state change in time, and RAM requires a Joules construction cost per bit storage space in cm^3. Of course RAM has state-change energy cost and CPU has construction cost, but those energy costs are smaller.
All economic schools have said following a basket of commodities is the best currency. Those favoring gold do so only because it is the best single commodity that has the right properties. It also one of the most wasteful ways to use kinetic energy, which is key to its perceived value. A basket would require measurements and "paper" (bits on a computer). The cost of energy (like electricity) is the largest underlying factor in the cost of producing commodities. So currencies based on Joules have been proposed as ideal. Zcash is a Joules-based currency. The Joules-as-value, both kinetic and potential, has a deep connection with computation and life. (see Note at bottom).
There is a 4th "benefit" to a high electrical cost per coin, although all these points are connected. It should not sell for less than the cost to produce it, unless someone has given up on the coin and will accept a loss.
Zcash's goal is to be "democratic" in mining. The result is an ideal cryptocoin. POW should not be "might is right", but "distributed might is right". Otherwise, the ability of miners to lobby the governing coders becomes the wrong kind of might.
This is not to say an energy-intensive coin is best for society. A coin that is given based on how much a person helps society (such as Solar Coin) would be best. But that involves agreement on definition of what is "best" (are solar cells really the best use of your money to be subsidized by giving you a coin?) and then measuring it before the cryptography part can even begin. It is a type of fiat requiring a middle man (or at least a group of oracles that are doing an agreed upon measurement, governed by smart contracts that define the rules for the distribution use of a specific coin). The whole reason fiat replaced gold is because governments are able to print it and distribute it evenly based on achieving goals that are ostensibly best for that society. Coins distributed based on POW that is not connected with the betterment of society are not best unless the government is not acting in the best interest of people and/or anarchy (e.g., hyperinflation) is near.
Note: To be more correct, it is the Joules as measured by Gibb's Free Energy that is essential to life. Schrodinger even updated his "What Is Life?" book to point out that when he said "negative entropy" as being the key to life, he really meant free energy. Gibb's F.E. = U -TS where U is internal energy and TS is temp x entropy. In terms of Zcash, U=RAM+CPU capital one-shot "energy" expense and TS=CPU operating on-going energy expense. The first is energy stored in spacial RAM and CPU structures, and the second is energy spent in time. CPU computations come at an energy cost of TS/eff where eff is the efficiency of the device. This does not include algorithm efficiency. Per N bits that change state irreversibly, the number of Joules expended is T x S / eff (see Landauer's limit relating bits to physical entropy) where S=kb x ln(2) x N. For a given algorithm.
Thursday, August 4, 2016
Note to self: cryptocurrency difficulty settings
I estimated above that the difficulty setting's averaging of the past in order to determine if coin production is on track or off-track should be 3 x N / SQRT(N). It's almost a perfect curve fit for a coin's expected deviation from a Poisson distribution, allowing for up to 3.75 standard deviations from the Poisson distribution's expected mean. This high level of permission allows for network hiccups away from the mean that someone could profit from if they can cost-effectively shift hashing energy around to different time slots. They'll be able to detect a deviation with decent probability each hour (N=30 rule) before the code decides in a difficulty change.
Poisson distribution with 3.75 std devs from mean:
3.75 x 2 e^-N x N^(N+1) / N! =~ 3 x N / SQRT(N)
If you want to minimize profit from hiccups, you could remove the 3.75 to allow for 1 std dev from the mean. The drawback is that this means that 1/3 of the time you will be intervening with a change in difficulty where none was statistically present, instead of ~0.01% of the time with 3.75. 3.75 is too permissive.
With the current method, the algorithm appears to be intervening too much with too-large changes that are too-often. It seems like a nosy government regulator, acting beyond what the statistics requires. It is donating easy coins to slow transaction periods at the expense of the small business owner (miner), to the benefit of the more-talented, conglomerated smart businesses selling a service (shifting hash energy) to the slow transaction periods. I would use 2 std devs instead of 3.75 as a nod to statistical tradition. The current code is using something like 0.1. [edit correction: after N=6 intervals
It's not merely hiccups it's trying to fix, but also the distribution itself allows for a sparse period of transactions. The Poisson distribution says the probability of k=1 occurrence in N=6 intervals (15 minutes) is (N L)^k / k! / e^(N L) = N/e^N = 1.5% where L = the average of 1/2.5 minutes, so NL= average occurrences in N x 2.5 min intervals. So there will be an average wait of 1/0.015=67 intervals for it to take 15 minutes. It would take 30 minutes once every 23 days.
Poisson distribution with 3.75 std devs from mean:
3.75 x 2 e^-N x N^(N+1) / N! =~ 3 x N / SQRT(N)
If you want to minimize profit from hiccups, you could remove the 3.75 to allow for 1 std dev from the mean. The drawback is that this means that 1/3 of the time you will be intervening with a change in difficulty where none was statistically present, instead of ~0.01% of the time with 3.75. 3.75 is too permissive.
With the current method, the algorithm appears to be intervening too much with too-large changes that are too-often. It seems like a nosy government regulator, acting beyond what the statistics requires. It is donating easy coins to slow transaction periods at the expense of the small business owner (miner), to the benefit of the more-talented, conglomerated smart businesses selling a service (shifting hash energy) to the slow transaction periods. I would use 2 std devs instead of 3.75 as a nod to statistical tradition. The current code is using something like 0.1. [edit correction: after N=6 intervals
It's not merely hiccups it's trying to fix, but also the distribution itself allows for a sparse period of transactions. The Poisson distribution says the probability of k=1 occurrence in N=6 intervals (15 minutes) is (N L)^k / k! / e^(N L) = N/e^N = 1.5% where L = the average of 1/2.5 minutes, so NL= average occurrences in N x 2.5 min intervals. So there will be an average wait of 1/0.015=67 intervals for it to take 15 minutes. It would take 30 minutes once every 23 days.
Thursday, July 21, 2016
an idea for an ideal cryptocurrency
Introduction, context:
Previously I have discussed the problems with limited-quantity coins. This is a coin idea that might be constant-value, or half-way between. This is an outline (without addressing implementation problems) of what I think would be close to an ideal based on the idea that maximizing median human happiness is the "meaning of human life" and the implicit goal humans should be assigning to the economic machine. I do not say "our" economic machine because we are not intentionally (let alone intelligently) cooperating to maximize the benefits we receive from the machine as whole. The "invisible hand" of competitive, cooperative, honest selfishness at the individual transaction level is not a God that makes everything better for our system as a whole, without diligent, intelligent, conscious effort at the system-wide level (such as government regulations for rule of law to encourage safe and honest transactions, and against monopolies and pollution). The prisoner's dilemma does not have a synergistic gain during cooperation unless the rules and goals of the game are intelligently known and enforced. My goal is to prevent evolutionary optimizations from mindless humans and mindless machines to sneak into our economic optimizations without regard to human happiness. But as can be seen from the following, maximum median human happiness might turn out to be equivalent to letting the machines rise, encouraging a decrease in human population. This could be painful in the short term like the black plague, but beneficial in the long term like the enlightenment. But the machines have enough wealth in efficiency that the process does not need to be painful.
Coin description:
Assign a fixed amount of coin to each person on planet. Co-join their "DNA" (not necessarily a retinal scan) and a 2-factor authorization device (random key generator based) as part of their private key(s). The number of coins each adult (25 and older) with unique DNA receives along with their 2-factor device is 1,000,000. Young people receive 100,000 per year from age 16 to 25.
Governance, fees:
If world is overpopulated, people will have less purchasing power. Pay-weighted voting might be a good balance between democracy (which can cause too many regulations, socialism, and overpopulation) and stake-weighted voting (which has problems from insiders and lobbies). Pay-weighted voting might be the only tax for implementing the governing laws. If it's not, then fees, interest, and rent should be collected by the government, targeting entities that are acting against the overall goal which is "maximum happiness per median person".
Constant quantity or constant value?
I can't decide if it should be constant quantity like this (increasing only with population), or if the government can be allowed to expand or contract coin supply based on a basket of commodities. Tracking a basket of commodities keeps prices and wages very stable and prevents boom/bust cycles. Today's financial games driving commodity prices away from supply and demand (Szabo wrote an article on this) do not help in tracking a basket of commodities. Maybe if a measurable-quantity coin(s) takes over the world, these games are not possible. (would it be harder to do fraudulent/stupid derivatives and keyboard credit that pretends to be real coin?) Government printing could be directed to reduce the effect of technological dis-employment. Constant-quantity coin could encourage dis-employment and thereby lead to reduced population and increase median happiness per person.
Coin is continually created, but population is not increasing as fast as productivity gains, so it is half-way between a constant-quantity coin and a constant-value coin.
Previously I have discussed the problems with limited-quantity coins. This is a coin idea that might be constant-value, or half-way between. This is an outline (without addressing implementation problems) of what I think would be close to an ideal based on the idea that maximizing median human happiness is the "meaning of human life" and the implicit goal humans should be assigning to the economic machine. I do not say "our" economic machine because we are not intentionally (let alone intelligently) cooperating to maximize the benefits we receive from the machine as whole. The "invisible hand" of competitive, cooperative, honest selfishness at the individual transaction level is not a God that makes everything better for our system as a whole, without diligent, intelligent, conscious effort at the system-wide level (such as government regulations for rule of law to encourage safe and honest transactions, and against monopolies and pollution). The prisoner's dilemma does not have a synergistic gain during cooperation unless the rules and goals of the game are intelligently known and enforced. My goal is to prevent evolutionary optimizations from mindless humans and mindless machines to sneak into our economic optimizations without regard to human happiness. But as can be seen from the following, maximum median human happiness might turn out to be equivalent to letting the machines rise, encouraging a decrease in human population. This could be painful in the short term like the black plague, but beneficial in the long term like the enlightenment. But the machines have enough wealth in efficiency that the process does not need to be painful.
Coin description:
Assign a fixed amount of coin to each person on planet. Co-join their "DNA" (not necessarily a retinal scan) and a 2-factor authorization device (random key generator based) as part of their private key(s). The number of coins each adult (25 and older) with unique DNA receives along with their 2-factor device is 1,000,000. Young people receive 100,000 per year from age 16 to 25.
Governance, fees:
If world is overpopulated, people will have less purchasing power. Pay-weighted voting might be a good balance between democracy (which can cause too many regulations, socialism, and overpopulation) and stake-weighted voting (which has problems from insiders and lobbies). Pay-weighted voting might be the only tax for implementing the governing laws. If it's not, then fees, interest, and rent should be collected by the government, targeting entities that are acting against the overall goal which is "maximum happiness per median person".
Constant quantity or constant value?
I can't decide if it should be constant quantity like this (increasing only with population), or if the government can be allowed to expand or contract coin supply based on a basket of commodities. Tracking a basket of commodities keeps prices and wages very stable and prevents boom/bust cycles. Today's financial games driving commodity prices away from supply and demand (Szabo wrote an article on this) do not help in tracking a basket of commodities. Maybe if a measurable-quantity coin(s) takes over the world, these games are not possible. (would it be harder to do fraudulent/stupid derivatives and keyboard credit that pretends to be real coin?) Government printing could be directed to reduce the effect of technological dis-employment. Constant-quantity coin could encourage dis-employment and thereby lead to reduced population and increase median happiness per person.
Coin is continually created, but population is not increasing as fast as productivity gains, so it is half-way between a constant-quantity coin and a constant-value coin.
Wednesday, July 20, 2016
Proof of stake correlate with vote power, at least not anonymously
Post to github
Voting by POS is like lobbyists influencing the government. It protects or increases assets without regard to the long term health of the economy. For a coin, "health" means "used in the marketplace" or "long term value" (which are probably mutually dependent). These are abstract ideas too far away for the vast majority of stake holders who are looking at their bank account. Stock holders are biased towards short-term profit at the long-term expense of the company. Laws had to be put in place to stop it: public disclosure of who the largest stakeholders are and how much they have. The rest of the shareholders demand they restrict themselves to if and when they can cash out. These are not even options for Zcash, so voting by number of shares held as in stocks is not a good idea.
Paying to vote would be better, like the largest taxpayers deciding where government spends its money. Government is no more than code (law) directing the proper use a currency (legal tender for taxes, court, and the marketplace). People in this space need to realize coders=government, stakeholders=voters, and nodes=banks. Voters pay taxes to the government and fees to the banks. The only difference is that computers are infinitely more efficient and reliable than brains.
A short-term investor is less likely to spend money to vote. Money gained from the vote should be invested to increase the value of the coin (like taxes for gov). Helping merchants accept it is an obvious possibility, as is paying more to researchers/coders. Destroying the voting coins is possible, but limited quantity coins are already enough of a marketplace disaster that I would not encourage it. (Markets need constant value in order to keep contracts, wages, and prices valid in the same way science needs units and conversion factors to stay the same. )
Coders/researchers/economists should be the law makers, like Plato's elite philosophers, in designing the constitution. Voters can work out details and future problems. The goal is to make the game fun, fair, and long-lasting. The winning team should not be given extra power to change the rules in the middle of the game. The winning team should even be penalized so that the games remains a distributive network, unless they want to end up playing by themselves, with no external value.
=====
My sensitivity to seeing it [coin issuance curve changes] is partly based on decision-makers being large holders of the coin in quantities unknown even to each other. The rest of this paragraph is against this type of proof-of-stake voting, an issue I posted on yesterday #1112 The complexity of the tech issues in Zcash makes the founders de facto stock-like insiders despite it being open source, insiders that may have a preference for short-term gain at long term expense and not regulated to prevent this expectation. Explicit voting by size of stake is bad because it is a bias for short term holding value at expense of long-term marketplace-use value, which is the basis of long term holding value. Under the pretense of having an unbiased interest in the long-term health of the coin is worse. Proof of stake voting with noble pretenses underlies Bitcoin's woes. "Our developers will not mine". But how will you know? I am not really concerned about this major concern, but the laws have not caught up and the company could accidentally do things that would normally be illegal if it follows a path that is intuitively good. Voting by size of stake is a bias towards a chain letter. Doing it secretly is a bias towards ponziness. Does Satoshi's abandonment of the project indicate awareness of a conflict of interest as a large anonymous stakeholder / insider that should be illegal? Would him selling without disclosure normally be illegal and this is the reason he has not sold any?
All things considered, I think the company should explicitly state its contract with society (what the coin must always be) in its principles of organization, self-referentially unmodifiable, with a copy-left inheritance requirement in the event of a buyout, rigidly connected to and defining the "Zcash"
trademark. The target audience of the contract would be with future holders of the coin, not current holders. Then add it to the blog and code, before launch.
ETH giving release names seems more necessary because the system's philosophy and understanding of itself is still changing. This is why I have very little interest in it. I would like an asset. I want them to succeed in replacing government and banking. But I still want an asset I can understand that is not connected to complexity and self-identity confusion, let alone "Turing-complete security holes".
Both supply curve and name changes give an impression of "instability". Name changing is more of an issue with me because it implies the coin's identity is changing. That's great for improving products who's primary features are changing. They need to change identity in order to advance. But Zcash should have a rigid, limited, stated philosophical identity like Excel and Powerpoint instead of CPU-like name changes. I think Zcash is trying to be anonynous, secure, bitcoin quantity and at least similar curve, distributed mining as much as possible, and fast and efficient as much as possible. Since these features should only improve and without substantial change or any foreseeable addition, names seem to add only confusion as to what the name means (is it one of many products under a Zcash company umbrella? Has the coin changed it's anonymizing or hash method?) and gives an impression of changeability. Of course everyone wants the product to improve the stated goals, but not to otherwise change. A major anonymizing or hash algorithm change is a detail that should fall under Zcash "2.0" or whatever release notes.
Voting by POS is like lobbyists influencing the government. It protects or increases assets without regard to the long term health of the economy. For a coin, "health" means "used in the marketplace" or "long term value" (which are probably mutually dependent). These are abstract ideas too far away for the vast majority of stake holders who are looking at their bank account. Stock holders are biased towards short-term profit at the long-term expense of the company. Laws had to be put in place to stop it: public disclosure of who the largest stakeholders are and how much they have. The rest of the shareholders demand they restrict themselves to if and when they can cash out. These are not even options for Zcash, so voting by number of shares held as in stocks is not a good idea.
Paying to vote would be better, like the largest taxpayers deciding where government spends its money. Government is no more than code (law) directing the proper use a currency (legal tender for taxes, court, and the marketplace). People in this space need to realize coders=government, stakeholders=voters, and nodes=banks. Voters pay taxes to the government and fees to the banks. The only difference is that computers are infinitely more efficient and reliable than brains.
A short-term investor is less likely to spend money to vote. Money gained from the vote should be invested to increase the value of the coin (like taxes for gov). Helping merchants accept it is an obvious possibility, as is paying more to researchers/coders. Destroying the voting coins is possible, but limited quantity coins are already enough of a marketplace disaster that I would not encourage it. (Markets need constant value in order to keep contracts, wages, and prices valid in the same way science needs units and conversion factors to stay the same. )
Coders/researchers/economists should be the law makers, like Plato's elite philosophers, in designing the constitution. Voters can work out details and future problems. The goal is to make the game fun, fair, and long-lasting. The winning team should not be given extra power to change the rules in the middle of the game. The winning team should even be penalized so that the games remains a distributive network, unless they want to end up playing by themselves, with no external value.
=====
My sensitivity to seeing it [coin issuance curve changes] is partly based on decision-makers being large holders of the coin in quantities unknown even to each other. The rest of this paragraph is against this type of proof-of-stake voting, an issue I posted on yesterday #1112 The complexity of the tech issues in Zcash makes the founders de facto stock-like insiders despite it being open source, insiders that may have a preference for short-term gain at long term expense and not regulated to prevent this expectation. Explicit voting by size of stake is bad because it is a bias for short term holding value at expense of long-term marketplace-use value, which is the basis of long term holding value. Under the pretense of having an unbiased interest in the long-term health of the coin is worse. Proof of stake voting with noble pretenses underlies Bitcoin's woes. "Our developers will not mine". But how will you know? I am not really concerned about this major concern, but the laws have not caught up and the company could accidentally do things that would normally be illegal if it follows a path that is intuitively good. Voting by size of stake is a bias towards a chain letter. Doing it secretly is a bias towards ponziness. Does Satoshi's abandonment of the project indicate awareness of a conflict of interest as a large anonymous stakeholder / insider that should be illegal? Would him selling without disclosure normally be illegal and this is the reason he has not sold any?
All things considered, I think the company should explicitly state its contract with society (what the coin must always be) in its principles of organization, self-referentially unmodifiable, with a copy-left inheritance requirement in the event of a buyout, rigidly connected to and defining the "Zcash"
trademark. The target audience of the contract would be with future holders of the coin, not current holders. Then add it to the blog and code, before launch.
ETH giving release names seems more necessary because the system's philosophy and understanding of itself is still changing. This is why I have very little interest in it. I would like an asset. I want them to succeed in replacing government and banking. But I still want an asset I can understand that is not connected to complexity and self-identity confusion, let alone "Turing-complete security holes".
Both supply curve and name changes give an impression of "instability". Name changing is more of an issue with me because it implies the coin's identity is changing. That's great for improving products who's primary features are changing. They need to change identity in order to advance. But Zcash should have a rigid, limited, stated philosophical identity like Excel and Powerpoint instead of CPU-like name changes. I think Zcash is trying to be anonynous, secure, bitcoin quantity and at least similar curve, distributed mining as much as possible, and fast and efficient as much as possible. Since these features should only improve and without substantial change or any foreseeable addition, names seem to add only confusion as to what the name means (is it one of many products under a Zcash company umbrella? Has the coin changed it's anonymizing or hash method?) and gives an impression of changeability. Of course everyone wants the product to improve the stated goals, but not to otherwise change. A major anonymizing or hash algorithm change is a detail that should fall under Zcash "2.0" or whatever release notes.
Saturday, July 16, 2016
A.I. and economics again, post to zcash forum
Your 1st and 2nd sentences seem contradictory. My view is that making ASIC infeasible (to level the playing field) is a drastic market interference, for a good reason. A free market evolves towards concentration of wealth and monopolies. Democratic voting creates a more level playing field (1 person = 1 vote) by causing government to write the rules (algorithmic protocol) to bias the free market (capitalism) away from concentration of wealth, towards socialization.
Developers are the "governing employees" that make Zcash more democratic, more social. Equal access to coins based on investment expense is a fair market, made possible by a "government" (algorithm). It's a democratic idea, 1 vote = an equal investment expense.
It's true my ROI is entirely speculative. My point was to show small miners will lose only if big miners lose. Equihash is a good system for preventing wealth concentration at the outset: if ASICs were feasible, special interests could be a problem at the outset. Look at bitcoin's miners.
The economic/democratic problems I'm about to describe for any constant-quantity coin like Zcash are long-term. Given no other option, not even in theory, I'm choosing Zcash to be in the 1% instead of the 99%. All economic woes are a consequence of the physics of evolution. There is no solution. Humans are not capable of subverting the physics of evolutionary progress towards higher efficiency.
Anonymity that prevents government from unjustly targeting individuals is a form of wealth distribution. It can take away power of special interests who try to subvert democracy. But it can also prevent government from performing the good aspects of its democratic role. Among other things, if a constant quantity coin becomes the default currency, compound interest always results in wealth concentration in the lenders. Gold historically works only in times of anarchy and war. The people needing loans also need an inflating currency (but not inflationary prices). Ideally all the interest charges should be used to finance all of government. Interest should be the only tax, and that tax should fund the expansion of the society (which is ~ equal to its need for the currency) so that there is no inflation in prices or wages, which keeps contracts in that coin valid (think ETH).
Equitable computation is a more intelligent network (solves problems) for deep reasons, despite being less efficient. In A.I., the most effective systems evenly distribute computation. The constant quantity of total available CPU time and memory space is the "currency" that needs to be distributed to grant access. There's a conversion factor between the CPU time and memory space that is not unrelated to Einstein's meters=i*c*seconds based on Landauer principle. Genetic algorithms, agoric economic agent systems, bayesian techniques, & neural nets seek to redistribute computation among a wider variety of "genes/agents/nodes" and the "weighting factors" ("wiring" or "links" in the web) between them by distributing computational requirements more evenly, economizing the resources towards solutions. An unused node, gene, or web page (no links to it) and a very low price (in agoric agent-based computing) or very low probability (bayesian) are all computational elements that can be eliminated from the algorithm with minor error (a universal NAND gate with no wiring to it is the simplest example of an unused computational element).
Like everyone else interested in cryptocurrencies, I want to make as much profit with the least amount of work. Constant quantity currencies might be ideally suited for the 1% and a subversion of democracy. Wei Dai expressed a similar concern about bitcoin. That's why he likes tracking commodities. An ideal coin would expand in lock-step with its M2-like usage to keep wages and prices constant, which keeps contracts in that coin valid, and prevents early adopters from profiting (gaining more access to society's finite resources) without having to work for it (contributing to society).
A constant quantity currency is contrary to equitable (intelligent) economics for these reasons. It is only optimal when the resources it represents control of are a constant, as in A.I. systems constrained a specific hardware system. It will be beneficial in times of war and anarchy for the survivalists have planned ahead. Anonymity amplifies this benefit.
Sunday, May 29, 2016
The moral intelligence of Japanese greed
This was a post in response to an article describing Subaru's comeback that included marketing to lesbians in the late 1990's. The article: http://priceonomics.com/how-an-ad-campaign-made-lesbians-fall-in-love-with/
Not being offended by lesbianism and accepting it as just a normal part of life, i.e. not even being aware it is even a "thing", bespeaks of an intelligence and open society. Combined with durability, safety, and practicality, there was a larger group of intelligent people Subaru was appealing to than just the 5 micro-groups mentioned in the article. These qualities are very Japanese. For some reason, I always thought of Subaru as European with an American flavor. It seems much more American to me than Nissan, Mitsubishi, Toyota, or Honda. Seemingly unique and different has helped them. Everyday I see a neighbor's Subaru and think "how intelligent and different" and I am a little jealous that I don't own a Subaru. He parks the car backwards in the driveway which is done in case a quick exit is needed, a mark of intelligence and concern for safety, which seems to be the features Subaru exhibits and attracts. Lack of a male companion and being an "outcast" (at least in the past) possibly makes lesbians more concerned about safety in general, not just dependability. It's an intriguing story that goes beyond lesbianism. It's kind of distracting that it's cast as a "controversial" topic. There's something more here than gay rights, marketing, or controversy, as presented in the article. I think it's a triumph stemming from the Japanese people being simple, rational, and non-judgmental. If only the rational pursuit of profit were always like this.
Not being offended by lesbianism and accepting it as just a normal part of life, i.e. not even being aware it is even a "thing", bespeaks of an intelligence and open society. Combined with durability, safety, and practicality, there was a larger group of intelligent people Subaru was appealing to than just the 5 micro-groups mentioned in the article. These qualities are very Japanese. For some reason, I always thought of Subaru as European with an American flavor. It seems much more American to me than Nissan, Mitsubishi, Toyota, or Honda. Seemingly unique and different has helped them. Everyday I see a neighbor's Subaru and think "how intelligent and different" and I am a little jealous that I don't own a Subaru. He parks the car backwards in the driveway which is done in case a quick exit is needed, a mark of intelligence and concern for safety, which seems to be the features Subaru exhibits and attracts. Lack of a male companion and being an "outcast" (at least in the past) possibly makes lesbians more concerned about safety in general, not just dependability. It's an intriguing story that goes beyond lesbianism. It's kind of distracting that it's cast as a "controversial" topic. There's something more here than gay rights, marketing, or controversy, as presented in the article. I think it's a triumph stemming from the Japanese people being simple, rational, and non-judgmental. If only the rational pursuit of profit were always like this.
Wednesday, May 18, 2016
Benford's law, Zipf's law, and Yule-Simon distribution
Summary:
Language and population drop off at both ends from the log-log plot.
Benford's law is better than Zipf's for population and language, capturing the most common words better. It's below the log-log on the front end compared to Zipf's. But it seems sensitive to change.
Yule-Simon is best in the sense that it has an algebraic function that is easily solvable and is better than Zipf's, dropping off at the high on a log-log plot as is seen in population and language. It is based on evolution, I believe considering new species being added. When made "with memory" (not so algebraic, probably a differential equation), it was made to work really good. It might apply really well to social/computer networks where nodes are added. Words have connections to each other like a network.
Douple Pareto Log-Normal (DPLN) seems to have more interest, maybe even applicable to a lot of physics. It combines "geometric Brownian motion" (GBM) (a differential equation with a feed source and random changes) and Yule-Simon. The GBM is a "pure form" of Gibrat's law for cities. Gibrat's says cities start with a log normal distribution, which I believe causes the tail end to drop off since Yule drops of the other end. Pareto is log-log and has a "killing constant" that might drop off the tail. I do not know why they call it double pareto unless it is because it is like using two pareto curves, one for the top and one for the bottom.
The differential equations seem to be needed because it allows a "feedback", i.e. current state is used to calculating future states. For example, words, species, and cities are competing with each other for popularity in a limited "space". People feed words by employing them, environment feeds (employs) species, and cities employ (feed) people. But once feeding gets massive, there is drawback: the more a word is used, the less information it can convey due to how Shannon entropy/word is calculated. City density starts decreasing the efficiency benefits. Environments run out of food. On the tail end, rare words carry a lot of info, but few people know them. Fewer members of a species means less mating opportunities for gains in natural selection (Darwin realized this). Fewer people means fewer job options. There is a wide middle ground with an exponential. It is higher on the tail end as "population" benefit starts to kick in, and decreases at the high end as efficiency exponential starts being blocked by the energy (species), time (language), or spatial (cities) limits.
This is possibly my favorite of the articles:
http://www.cs.uml.edu/~zfang/files/dpln_ho.pdf
I checked Benford's law log(1+1/r) times 2.2 compared to Mandelbrot's modified Zipf law ~1/(r+2.7) for english. After the rank of 21, the error is less than 5%. It's higher for ranks 1 to 21, matching the first few English words better. Both are too high for large r. Benford also predicts country populations better.
Concerning the relationship between the Zipf and Benford:
http://mathworld.wolfram.com/BenfordsLaw.html
The Pareto distribution is a similar function applied to wealth (Xmin/X)^a where a greater than 1 and has been used as a measure of wealth inequality.
But it appears the wide-ranging real-world observations of these power-like laws is largely the result of the "preferential attachment". In short "success breeds success", the rich get richer. Words that are common become more common because they are common. Same thing with cities and species. Darwin wrote about how species become distinct because when you have a larger population to breed with, you have more options for the best selecting the best. Cities become more efficient in terms of providing potential employment. Companies gain efficiency as they get larger, allowing them to get larger. The kind of ranking that results from this is the Yule-Simon distribution. On a log-log plot, it give the most common words lower than expected from a log-log plot, which is what words do. It's formula is
freq = x*x!R!/(x + R)!
where x! is the gamma function of x+1 and x is a real value greater than 0. R = rank-1. (x+R)! is the gamma function of (x+1+R). The Gamma function is the continuous version of (N-1)!. I would call x the "amplifier" in the positive feedback. k*k!*x!/(k+x)! For x=1 it is R!/(1+R)! = 1/R = zipf's law.
But it is inadequate for the tail end as it is straight when it also needs to drop off. One of the following papers used the formula expressed as P(r) = 1/r^a where a=1+1/(1-p) where p is a constant probability of a new word being added during a time step. In this version they modified it to have a downward concave shape, so it worked really well.
It has been show to model language excellently and in city population
Yule-simon Works better in language
Language and population drop off at both ends from the log-log plot.
Benford's law is better than Zipf's for population and language, capturing the most common words better. It's below the log-log on the front end compared to Zipf's. But it seems sensitive to change.
Yule-Simon is best in the sense that it has an algebraic function that is easily solvable and is better than Zipf's, dropping off at the high on a log-log plot as is seen in population and language. It is based on evolution, I believe considering new species being added. When made "with memory" (not so algebraic, probably a differential equation), it was made to work really good. It might apply really well to social/computer networks where nodes are added. Words have connections to each other like a network.
Douple Pareto Log-Normal (DPLN) seems to have more interest, maybe even applicable to a lot of physics. It combines "geometric Brownian motion" (GBM) (a differential equation with a feed source and random changes) and Yule-Simon. The GBM is a "pure form" of Gibrat's law for cities. Gibrat's says cities start with a log normal distribution, which I believe causes the tail end to drop off since Yule drops of the other end. Pareto is log-log and has a "killing constant" that might drop off the tail. I do not know why they call it double pareto unless it is because it is like using two pareto curves, one for the top and one for the bottom.
The differential equations seem to be needed because it allows a "feedback", i.e. current state is used to calculating future states. For example, words, species, and cities are competing with each other for popularity in a limited "space". People feed words by employing them, environment feeds (employs) species, and cities employ (feed) people. But once feeding gets massive, there is drawback: the more a word is used, the less information it can convey due to how Shannon entropy/word is calculated. City density starts decreasing the efficiency benefits. Environments run out of food. On the tail end, rare words carry a lot of info, but few people know them. Fewer members of a species means less mating opportunities for gains in natural selection (Darwin realized this). Fewer people means fewer job options. There is a wide middle ground with an exponential. It is higher on the tail end as "population" benefit starts to kick in, and decreases at the high end as efficiency exponential starts being blocked by the energy (species), time (language), or spatial (cities) limits.
This is possibly my favorite of the articles:
http://www.cs.uml.edu/~zfang/files/dpln_ho.pdf
I checked Benford's law log(1+1/r) times 2.2 compared to Mandelbrot's modified Zipf law ~1/(r+2.7) for english. After the rank of 21, the error is less than 5%. It's higher for ranks 1 to 21, matching the first few English words better. Both are too high for large r. Benford also predicts country populations better.
Concerning the relationship between the Zipf and Benford:
http://mathworld.wolfram.com/BenfordsLaw.html
The Pareto distribution is a similar function applied to wealth (Xmin/X)^a where a greater than 1 and has been used as a measure of wealth inequality.
But it appears the wide-ranging real-world observations of these power-like laws is largely the result of the "preferential attachment". In short "success breeds success", the rich get richer. Words that are common become more common because they are common. Same thing with cities and species. Darwin wrote about how species become distinct because when you have a larger population to breed with, you have more options for the best selecting the best. Cities become more efficient in terms of providing potential employment. Companies gain efficiency as they get larger, allowing them to get larger. The kind of ranking that results from this is the Yule-Simon distribution. On a log-log plot, it give the most common words lower than expected from a log-log plot, which is what words do. It's formula is
freq = x*x!R!/(x + R)!
where x! is the gamma function of x+1 and x is a real value greater than 0. R = rank-1. (x+R)! is the gamma function of (x+1+R). The Gamma function is the continuous version of (N-1)!. I would call x the "amplifier" in the positive feedback. k*k!*x!/(k+x)! For x=1 it is R!/(1+R)! = 1/R = zipf's law.
But it is inadequate for the tail end as it is straight when it also needs to drop off. One of the following papers used the formula expressed as P(r) = 1/r^a where a=1+1/(1-p) where p is a constant probability of a new word being added during a time step. In this version they modified it to have a downward concave shape, so it worked really well.
It has been show to model language excellently and in city population
Yule-simon Works better in language
works better in cities
But there is a dropping off of the log-log straight line at both ends in most data that the straight Yule-Simon law does not handle. Successful cities do not merely add new nearby cities as Yule shows. The bigger city's relative population drops off from from this which is a different way of saying maybe overpopulation starts losing efficiency of its attraction. On the tail end there is there are otehr disadvantages. Commonly-used words are used more often because they are common, but since they convey less information due to being common, the effect is limited which prevents it from following a straight log-log curve. On the other end rare words are more rare than expected because not enough people know them to be able to usually use them. Similarly cities would follow a strict log-log curve due to statistics, but inefficiencies are created for different reasons in the most and least populated regions. In animals, they either start eating each other's food source, or they are not able to find a mate. Wealth on the other hand may not be subject to an "overpopulation" effect.
So the DPLN may be the ultimate:
For cities if not a wide range of physics, it seems better to combine Yule with the Geometric Brownian Motion (GBM, random variation of a random variable with a fuel source for new entrants) which is supposed to be Gibrat's log-normal law for cities in its pure form.
"A random variable X is said to follow GBM if its behavior over time is governed by the following differential equation dX = (µdt +σdB)X, (15) where dB is the increment of a standard Brownian motion (a.k.a. the white noise). For a GBM the proportional increment of X in time dt comprises a systematic component µdt, which is a steady contribution to X, and a random component σdB, which is fluctuated over time. Thus the GBM can be seen to be a stochastic version of simple exponential growth."
GBM feeds in new populations or words, and where they settle has a random fluctuation. Maybe this some how causes the tail to drop off, as Yule causes the high end to drop off.
Here's the best complete explanation of city sizes.
"The double Pareto lognormal seems more appropriate since it comprises a lognormal body and power law tails. Reed [36] suggests a GBM model, similar to the one that models personal incomes, for obtaining the settlement size distribution. Individual human settlements grow in many different ways. At the macro level a GBM process can be used to model the size growth by assuming a steady systematic growing rate and a random component. The steady growing rate reflects the average growth rate over all settlements and times, and the random component re- flects the variability of the growth rate. The time when a city is founded varies from settlement to settlement. If we assume in the time interval (t,t + dt) any existing settlement can form a new satellite settlement with probability λdt, the creation of settlements is a Yule process [39], which was first proposed as a model for the creation of new biological species. Under Yule process, the expected number of settlements is e^λt after t time since the first settlement. That is, the number of settlements is growing at rate λ. Therefore, the existing time for all settlements is exponentially distributed. It is straightforward to conclude that under GBM and Yule processes, the overall settlements size distribution will is a double Pareto distribution. If we further assume a lognormal initial assume a lognormal initial settlement size, the result will converge to the double Pareto lognormal distribution
Reed 2004 , DPLN invention, applicable to physics
Thursday, May 12, 2016
Problem with bitcoin and gold as currency
In a previous post I discussed the problem with bitcoin's constant-quantity of money. Wei Dai has commented that he views bitcoin as probably problematic for probably similar reasons. But even an asset-backed currency such as Gold or b-money has a problem.
Hard core money, an objective asset that retains its value is great when doing transactions with potential enemies. It should have an important place in transactions across disparate legal systems such as countries. You want to walk away without future obligation (a threat). "Cash on the barrel head" has its place with potential enemies not mutually adhering to a higher law or assurance of mutual benefit after the transaction (that higher law and mutual benefit are often the same thing). But money does not have to be meant only to objectively optimize isolated transactions without regard to a wider society. It can be more intelligent than that, finding optimal solutions to the prisoner's dilemma on a grander scale, beyond the immediate participants in a transaction.
The problem (or lack of "optimality") occurs in systems where you are not the only one who is important to you. It's not ridiculous or anti-evolution-theory to assume you will sacrifice a part of your profit for the benefit of others, especially if it is a small cost to you and a great benefit to others. If you count your success as dependent on society's success and not just your bank balance, there's more to consider. This is why a constant-value coin is not ideal. By making the value of the asset vary with something other than a stable asset, pro-social law (aka system-wide intelligence) can be implemented.
The fundamental problem with a constant-quantity coin like Bitcoin or gold is that it is an anti-social money. It seeks for the holder to maintain value without regard to the condition of society. Society can go to hell and Gold (at least) will still have value. That's a-social. Past transactions that result in a person holding an asset should be post-invalidated if the sum of those transactions resulted in disaster for all. Every transaction should carry a concern for all, present and future. That is a characteristic of a system that displays cooperative intelligence. There should always be a feedback measurement from the future of the entire community of people you (should) care about back to your current wealth. This feedback is a scientific measurement as if the past was an experiment. It enforces decisions on how to make future measurements, seeking an optimal outcome. Defining an optimal outcome should be the first step. (this is not easy, see footnote 1). Deciding how to measure it is the second step. Deciding how to use the measurement in order to adjust your actions in order to maximum the outcome is the core intelligence (see footnote 2), once you've realized the important of steps 1 and 2. Technology has advanced so rapidly, we never formalized a consensus goal for 1 well enough for defining a number 2. As Einstein said, the defining characteristic of our age is an excess of means without knowing what we want. It used to be we just wanted money so that we could have food, sex, and children, or to have enough pride via money and/or social relative to our peers that we felt justified in having children.
Side note, Nick Szabo has pointed out that keyboard credit from modern banking allows speculators to change the price of commodities as much as supply and demand. In what I describe here, that would need to be prevented.
This is why a coin that adjusts to keep commodity prices constant is more intelligent. Laws against monopolies and pollution can regulate transactions to prevent the anti-social nature of maximizing profit per transaction. That's not the benefit of a commodity coin. A commodity-coin has a different kind of system-wide intelligence. If commodities are in excess to demand, the prices will try to fall. So a currency following a basket of commodities will "print" more of itself to keep commodity prices stable. In a growing economy, the excess money could replace taxes, so it would merely fund government, or it could fund the building of more infrastructure to make it's workers healthier, happier, and/or more competitive with other countries. That would demonstrate intelligence that is good for the system's future. A less intelligent outcome (defined as bad for the future strength of the system) is to print the money to buy off voters or to bailout corrupt, inefficient, useless banks with QE.
Printing more money when commodity prices falls prevents the type of destructive deflation that occurred in the Great Depression. Instead of printing more money, they burned food in the fields. They stopped producing real assets like commodities on purpose instead of producing paper money.
If commodities get scarce, the money supply would contract along with them, raising its value. This promotes savings and working. Theoretically the savings and working would be directed towards more commodity production to return the system to health.
In the first case, an economic boom is allowed because the availability of commodities indicated it could afford it. In the second case a bust is prevented by making everyone work harder.
In the first case, savers are penalized. It should be this way because their capital is no longer needed to invest in producing more commodities. It needs to be spent, and if they are not spending it, then "the people" will spend it on the street, reaping the rewards of past savings. Commodities are the measure because they are the fundamental inputs to everything else that require the largest investments.
In the second case, everyone is biased towards being more of a saver.
footnote 1) Should we have a higher median happiness, or a higher median happiness times number of people? Should we restrict it to people? Or should we have a preference for machines? They're infinitely more efficient (see past posts of my measurements of their ability to acquire energy, move matter, create strong structures, and to think about how to do it). They'll be the only ones capable of repelling an alien invasion and to engage in the most successful invasions themselves.
footnote 2) Intelligence requires feedback from observation to modify the algorithm. Engineering control theory writes it as a differential equation and block diagrams "consciousness" as the subtraction from where you are from where you want to be, and takes action on the difference (the error signal). I am not sure if there's any A.I. that is not a variation of this. If it is not making an observation (science) to increase intelligence, is it an adaptable intelligence? In past posts I've mentioned how the simplest form of this feedback is also an element (like a NAND or XOR gatee) that can be used to implement a complete Turing machine. A house thermostat is an example. There is also a reduction in entropy in intelligence, taking a lot of observation to classify observation into a much smaller set of action. The error-signal-of-consciousness may need a reduction (classification) of the observed world. I believe Schrodinger discussed this in "What is Life?"
Hard core money, an objective asset that retains its value is great when doing transactions with potential enemies. It should have an important place in transactions across disparate legal systems such as countries. You want to walk away without future obligation (a threat). "Cash on the barrel head" has its place with potential enemies not mutually adhering to a higher law or assurance of mutual benefit after the transaction (that higher law and mutual benefit are often the same thing). But money does not have to be meant only to objectively optimize isolated transactions without regard to a wider society. It can be more intelligent than that, finding optimal solutions to the prisoner's dilemma on a grander scale, beyond the immediate participants in a transaction.
The problem (or lack of "optimality") occurs in systems where you are not the only one who is important to you. It's not ridiculous or anti-evolution-theory to assume you will sacrifice a part of your profit for the benefit of others, especially if it is a small cost to you and a great benefit to others. If you count your success as dependent on society's success and not just your bank balance, there's more to consider. This is why a constant-value coin is not ideal. By making the value of the asset vary with something other than a stable asset, pro-social law (aka system-wide intelligence) can be implemented.
The fundamental problem with a constant-quantity coin like Bitcoin or gold is that it is an anti-social money. It seeks for the holder to maintain value without regard to the condition of society. Society can go to hell and Gold (at least) will still have value. That's a-social. Past transactions that result in a person holding an asset should be post-invalidated if the sum of those transactions resulted in disaster for all. Every transaction should carry a concern for all, present and future. That is a characteristic of a system that displays cooperative intelligence. There should always be a feedback measurement from the future of the entire community of people you (should) care about back to your current wealth. This feedback is a scientific measurement as if the past was an experiment. It enforces decisions on how to make future measurements, seeking an optimal outcome. Defining an optimal outcome should be the first step. (this is not easy, see footnote 1). Deciding how to measure it is the second step. Deciding how to use the measurement in order to adjust your actions in order to maximum the outcome is the core intelligence (see footnote 2), once you've realized the important of steps 1 and 2. Technology has advanced so rapidly, we never formalized a consensus goal for 1 well enough for defining a number 2. As Einstein said, the defining characteristic of our age is an excess of means without knowing what we want. It used to be we just wanted money so that we could have food, sex, and children, or to have enough pride via money and/or social relative to our peers that we felt justified in having children.
Side note, Nick Szabo has pointed out that keyboard credit from modern banking allows speculators to change the price of commodities as much as supply and demand. In what I describe here, that would need to be prevented.
This is why a coin that adjusts to keep commodity prices constant is more intelligent. Laws against monopolies and pollution can regulate transactions to prevent the anti-social nature of maximizing profit per transaction. That's not the benefit of a commodity coin. A commodity-coin has a different kind of system-wide intelligence. If commodities are in excess to demand, the prices will try to fall. So a currency following a basket of commodities will "print" more of itself to keep commodity prices stable. In a growing economy, the excess money could replace taxes, so it would merely fund government, or it could fund the building of more infrastructure to make it's workers healthier, happier, and/or more competitive with other countries. That would demonstrate intelligence that is good for the system's future. A less intelligent outcome (defined as bad for the future strength of the system) is to print the money to buy off voters or to bailout corrupt, inefficient, useless banks with QE.
Printing more money when commodity prices falls prevents the type of destructive deflation that occurred in the Great Depression. Instead of printing more money, they burned food in the fields. They stopped producing real assets like commodities on purpose instead of producing paper money.
If commodities get scarce, the money supply would contract along with them, raising its value. This promotes savings and working. Theoretically the savings and working would be directed towards more commodity production to return the system to health.
In the first case, an economic boom is allowed because the availability of commodities indicated it could afford it. In the second case a bust is prevented by making everyone work harder.
In the first case, savers are penalized. It should be this way because their capital is no longer needed to invest in producing more commodities. It needs to be spent, and if they are not spending it, then "the people" will spend it on the street, reaping the rewards of past savings. Commodities are the measure because they are the fundamental inputs to everything else that require the largest investments.
In the second case, everyone is biased towards being more of a saver.
footnote 1) Should we have a higher median happiness, or a higher median happiness times number of people? Should we restrict it to people? Or should we have a preference for machines? They're infinitely more efficient (see past posts of my measurements of their ability to acquire energy, move matter, create strong structures, and to think about how to do it). They'll be the only ones capable of repelling an alien invasion and to engage in the most successful invasions themselves.
footnote 2) Intelligence requires feedback from observation to modify the algorithm. Engineering control theory writes it as a differential equation and block diagrams "consciousness" as the subtraction from where you are from where you want to be, and takes action on the difference (the error signal). I am not sure if there's any A.I. that is not a variation of this. If it is not making an observation (science) to increase intelligence, is it an adaptable intelligence? In past posts I've mentioned how the simplest form of this feedback is also an element (like a NAND or XOR gatee) that can be used to implement a complete Turing machine. A house thermostat is an example. There is also a reduction in entropy in intelligence, taking a lot of observation to classify observation into a much smaller set of action. The error-signal-of-consciousness may need a reduction (classification) of the observed world. I believe Schrodinger discussed this in "What is Life?"
Tuesday, May 10, 2016
relation between language and physical entropy, the dimensions, zipf's law
A post to reddit on possibility of Wei Dai being Satoshi.
Thanks for the explanation. Languages have various "attractors" for letters, words, and words groupings (idioms). The letter frequencies are not random because they represent phonemes that come from physical mouths that must follow a certain forward sequence. Listen to words backwards and you know how hard it would be to say it like that and you can't recognize anything. Listen to music backwards where the instruments are time-symmetrical due to less complexity compared to a mouth and in 2 seconds you know the song and it has the same emotional content, minus the words.
People expect certain word sequences. The word and phoneme "attractors" are like a gravitational field in 2 or 3 dimensions. Someone smart and always writing instead of talking can break away from the phoneme and expectation attractors and convey a lot in a few words. Einstein was like this. Szabo has half the frequency of his most common words compared to Satoshi and Wei which means his language is changing more. There's more true information content. On the other hand, someone smart or always talking instead of writing may want to be very clear to everyone and not break convention.
The extent to which a person has attractors (is living in a strong gravitational field) determines how sharply their word frequency drops down (Zipf's law for words in language, city populations, etc). Closer to "earth" would be higher word frequency, or living in a high gravitational field forces more words closer to Earth. Szabo's intelligence (or lack of too much concern if you can't follow) allows him to escape the gravity and say rare words more often, conveying more information. Measuring that was my original objective. That could be an independent way to identify an author (it's a different single dimension metric that conflates all the word dimensions you're talking about into one).
Large cities have an attractor based on opportunity and efficiency caused by the concentration of people that's self-re-enforcing. Convention in a community is self-re-enforces in words. So is "ease of speaking vowels" so they occur more frequently because less real energy is required to speak them, so they are in a low gravitational potential.
*[edit: My point in all this is to point out the curse of dimensionality as I understand it from you is that it assumes a random distribution. In my view, the "upper atmosphere", although larger in volume per radius increase from the center (the metric we're interested in), there will be fewer gas particles per volume (words) due to the gravity of a speaking/writing human's constraints). Our objective is to identify constraints that all people have, but also have varying gravitational constants for that constraint. People have different nitrogen to oxygen atom ratios in their atmospeheres. I have strong interest and experience in the relation between physical and information entropy, and words are at the intersection. Everything is a word, aka a symbol on a Turing machine, and people are running different algorithms on those symbols. The physical entropy is a function of ln(states/N!) where N is the number of particles and words also have this ln(1/N!) characteristic due to zipf's law and both are related to an energy in the system. Normal Shannon entropy assumes sampling with replacement is possible (2^n =states where n= number of bits and N=2 unique symbols), but this is not the case in physical entropy where each particle is sampled only once, so (1/N!)^n shows up as well as in fixed-length text where people have constraints on how often they can or will choose a word. computers do not have this constraint because there is not an energy cost to sampling with replacement. ]*
The origins of Zipf's law has always been a mystery. Many remember reading about it in Murray Gell-Mann's the Quark and the Jaguar. It was the only interesting thing in his book. But recently there have been good papers showing how it is probably derivable from Shannon's entropy when each word or person has a a a log of the energy cost or energy savings by being attracted to local groupings. There's feedback going on, or a blocking which means y=y' in differential equations so that the sum (integral) of y=1/x (which is Zipf's law, x=rank, y=frequency) gives a ln(x). So we're not fundamentally checking frequencies as much as we're comparing the rank of each word by using ln(x1/x2) which a subtraction of a frequency ln(x1) - ln(x2). Actually, we might need to do this on ranking's instead of frequencies, but you can see how similar it is. I did try it briefly and did not notice a difference. But there may be some good idea like applying it to singles with the other method on pairs, then finding conversion factor multpilier between the two before adding them (or a sum of their squares which won't make much difference) for a similarity (or author difference) metric.
It's always better to use lower-dimensions when there is a real low number of dimensional attractors working behind the scenes, if you know how to rank "how high" each word, word pair, or vowel is in that dimension. It's best (at least less noisy) but difficult to remove the effect the other 2 dimensions are having, probably requiring something like bayes theorem. Stylometry (word diagramming) would be a 4th dimension. There is a real physical person that works differently in those dimension, so it not good to be reducing them to a single dimension. The animal organ weights are only rough. Placing each weight in a dimension and not conflating the dimensions gives infinitely better categorization. Each word could be a dimension like you say, based on someone's experience and education. But if they are reading each other's writing and "attracted" to certain words and pairs because they know the other one uses it (Dai, Yudowksy, Back, Finney, and Satoshi) it reduces the chances they will NOT say the Satoshi words, by "taking up space" in what could have been said differently.
But in every word, letter, and idiom that is not in the core of the topic at hand, the simpler dimensions could show up and be measured by this sum of surprisals method, but broken out into 3 dimensions instead of 1. The group that won the Netflix prize started in hyperplanes of dimensions, whatever that means.
The open software SVMLight is the best way to do what I'm attempting (there's a simple ranking option), but I'd rather exhaust my ideas before trying to figure out how to use it.
What you're calling a gaussian" is really only because of a bad sampling of files, or having a true match. Great sampling should try to PREVENT having a "gaussian" good match by forcing it into a linear increase.
There should be a way to reduce or increase words in #1 and #2 as a result of comparing #1 and #2. Then increase or decrease the remaining word ratios. Then compare again with the mystery file and a true match should get better while the less match gets worse. "He who is the enemy of my enemy is my friend" or "He who is my friend's enemy is my enemy." It should be applied blindly, not making a distinction between #1 and #2 and being symmetrical.
Word pairs gave me twice as much distinction between the ratios I am saying are the key (#3-#2)/(#2-#1) = 5 whereas single, triple, and quad words gave 2.5. This was comparing Dai, Yudowsky, and gwern, all from the lesswrong site, and commonyl showing up in the same threads. I used 2MB on each to Satoshi's 253 kb.
Entropy of an ideal gas of n particles is S = A*ln[(Volume of container)^n/n!] +B*ln[((Energy in container)/n!)^n)]. This different from information entropy that takes the form S = log((values/memory location)^n) = N * H. Physical entropy carries more information per particle than information entropy does per symbol because of the n! that comes from the particles being selectable only once where symbols can be re-used. This would normally mean less information was possible. But the number of unique symbols in physical entropy is the number of states per particle which increases if not all the particles are carrying the energy. In short, physical entropy can carry information in different ways that information entropy cant.
But language has some physical entropy aspects to it. We can say the same message in many different ways that uses a larger or smaller set of symbols. Information entropy assumes the symbols used in a message were the only symbols that were available.
There is a physical energy cost for the different words we use, and there is a container of constraints (custom and word ordering) on the things we can say.
=============
udate: in trying to carry the above possible connection further, I've failed:
language entropy
S= N*sum(-k/rank/N*log(k/rank/N) = [A log(1) + B log(2) + ...] - k/(n/2*(n/2+1))* log(k)
Where N is a total words, not unique words n that are equal to max rank.
The entropy of an ideal gas (Sakur-Tetrode equation) of N molecules (and probably any physical entropy) can be written as
S = C*log((internal energy/N!)^N) + D*log(volume^N/N!)
S=N * [ C log(U) + D log(V) - C log(N!) ] - D log(N!)
===========
Thanks for the explanation. Languages have various "attractors" for letters, words, and words groupings (idioms). The letter frequencies are not random because they represent phonemes that come from physical mouths that must follow a certain forward sequence. Listen to words backwards and you know how hard it would be to say it like that and you can't recognize anything. Listen to music backwards where the instruments are time-symmetrical due to less complexity compared to a mouth and in 2 seconds you know the song and it has the same emotional content, minus the words.
People expect certain word sequences. The word and phoneme "attractors" are like a gravitational field in 2 or 3 dimensions. Someone smart and always writing instead of talking can break away from the phoneme and expectation attractors and convey a lot in a few words. Einstein was like this. Szabo has half the frequency of his most common words compared to Satoshi and Wei which means his language is changing more. There's more true information content. On the other hand, someone smart or always talking instead of writing may want to be very clear to everyone and not break convention.
The extent to which a person has attractors (is living in a strong gravitational field) determines how sharply their word frequency drops down (Zipf's law for words in language, city populations, etc). Closer to "earth" would be higher word frequency, or living in a high gravitational field forces more words closer to Earth. Szabo's intelligence (or lack of too much concern if you can't follow) allows him to escape the gravity and say rare words more often, conveying more information. Measuring that was my original objective. That could be an independent way to identify an author (it's a different single dimension metric that conflates all the word dimensions you're talking about into one).
Large cities have an attractor based on opportunity and efficiency caused by the concentration of people that's self-re-enforcing. Convention in a community is self-re-enforces in words. So is "ease of speaking vowels" so they occur more frequently because less real energy is required to speak them, so they are in a low gravitational potential.
*[edit: My point in all this is to point out the curse of dimensionality as I understand it from you is that it assumes a random distribution. In my view, the "upper atmosphere", although larger in volume per radius increase from the center (the metric we're interested in), there will be fewer gas particles per volume (words) due to the gravity of a speaking/writing human's constraints). Our objective is to identify constraints that all people have, but also have varying gravitational constants for that constraint. People have different nitrogen to oxygen atom ratios in their atmospeheres. I have strong interest and experience in the relation between physical and information entropy, and words are at the intersection. Everything is a word, aka a symbol on a Turing machine, and people are running different algorithms on those symbols. The physical entropy is a function of ln(states/N!) where N is the number of particles and words also have this ln(1/N!) characteristic due to zipf's law and both are related to an energy in the system. Normal Shannon entropy assumes sampling with replacement is possible (2^n =states where n= number of bits and N=2 unique symbols), but this is not the case in physical entropy where each particle is sampled only once, so (1/N!)^n shows up as well as in fixed-length text where people have constraints on how often they can or will choose a word. computers do not have this constraint because there is not an energy cost to sampling with replacement. ]*
The origins of Zipf's law has always been a mystery. Many remember reading about it in Murray Gell-Mann's the Quark and the Jaguar. It was the only interesting thing in his book. But recently there have been good papers showing how it is probably derivable from Shannon's entropy when each word or person has a a a log of the energy cost or energy savings by being attracted to local groupings. There's feedback going on, or a blocking which means y=y' in differential equations so that the sum (integral) of y=1/x (which is Zipf's law, x=rank, y=frequency) gives a ln(x). So we're not fundamentally checking frequencies as much as we're comparing the rank of each word by using ln(x1/x2) which a subtraction of a frequency ln(x1) - ln(x2). Actually, we might need to do this on ranking's instead of frequencies, but you can see how similar it is. I did try it briefly and did not notice a difference. But there may be some good idea like applying it to singles with the other method on pairs, then finding conversion factor multpilier between the two before adding them (or a sum of their squares which won't make much difference) for a similarity (or author difference) metric.
It's always better to use lower-dimensions when there is a real low number of dimensional attractors working behind the scenes, if you know how to rank "how high" each word, word pair, or vowel is in that dimension. It's best (at least less noisy) but difficult to remove the effect the other 2 dimensions are having, probably requiring something like bayes theorem. Stylometry (word diagramming) would be a 4th dimension. There is a real physical person that works differently in those dimension, so it not good to be reducing them to a single dimension. The animal organ weights are only rough. Placing each weight in a dimension and not conflating the dimensions gives infinitely better categorization. Each word could be a dimension like you say, based on someone's experience and education. But if they are reading each other's writing and "attracted" to certain words and pairs because they know the other one uses it (Dai, Yudowksy, Back, Finney, and Satoshi) it reduces the chances they will NOT say the Satoshi words, by "taking up space" in what could have been said differently.
But in every word, letter, and idiom that is not in the core of the topic at hand, the simpler dimensions could show up and be measured by this sum of surprisals method, but broken out into 3 dimensions instead of 1. The group that won the Netflix prize started in hyperplanes of dimensions, whatever that means.
The open software SVMLight is the best way to do what I'm attempting (there's a simple ranking option), but I'd rather exhaust my ideas before trying to figure out how to use it.
What you're calling a gaussian" is really only because of a bad sampling of files, or having a true match. Great sampling should try to PREVENT having a "gaussian" good match by forcing it into a linear increase.
There should be a way to reduce or increase words in #1 and #2 as a result of comparing #1 and #2. Then increase or decrease the remaining word ratios. Then compare again with the mystery file and a true match should get better while the less match gets worse. "He who is the enemy of my enemy is my friend" or "He who is my friend's enemy is my enemy." It should be applied blindly, not making a distinction between #1 and #2 and being symmetrical.
Word pairs gave me twice as much distinction between the ratios I am saying are the key (#3-#2)/(#2-#1) = 5 whereas single, triple, and quad words gave 2.5. This was comparing Dai, Yudowsky, and gwern, all from the lesswrong site, and commonyl showing up in the same threads. I used 2MB on each to Satoshi's 253 kb.
Entropy of an ideal gas of n particles is S = A*ln[(Volume of container)^n/n!] +B*ln[((Energy in container)/n!)^n)]. This different from information entropy that takes the form S = log((values/memory location)^n) = N * H. Physical entropy carries more information per particle than information entropy does per symbol because of the n! that comes from the particles being selectable only once where symbols can be re-used. This would normally mean less information was possible. But the number of unique symbols in physical entropy is the number of states per particle which increases if not all the particles are carrying the energy. In short, physical entropy can carry information in different ways that information entropy cant.
But language has some physical entropy aspects to it. We can say the same message in many different ways that uses a larger or smaller set of symbols. Information entropy assumes the symbols used in a message were the only symbols that were available.
There is a physical energy cost for the different words we use, and there is a container of constraints (custom and word ordering) on the things we can say.
=============
udate: in trying to carry the above possible connection further, I've failed:
language entropy
S= N*sum(-k/rank/N*log(k/rank/N) = [A log(1) + B log(2) + ...] - k/(n/2*(n/2+1))* log(k)
Where N is a total words, not unique words n that are equal to max rank.
The entropy of an ideal gas (Sakur-Tetrode equation) of N molecules (and probably any physical entropy) can be written as
S = C*log((internal energy/N!)^N) + D*log(volume^N/N!)
S=N * [ C log(U) + D log(V) - C log(N!) ] - D log(N!)
===========
An encoding scheme of a language when the language does NOT follow Zipf's law might result in the encoding following Benford's law (aka ~ Zipf's law). It might follow Benford's law better than most languages.
Language might follow Benford's law (data is more likely to begin with the number "1") instead of Zipf's law. I read English follows 1/rank0.85. In looking at the 1st table in the wolfram link below, I see Benford's law for rank 1 divided by rank 9 is almost exactly equal to saying English follows 1/rank0.85. Notice Benford's law is derived from a p(x)=1/x that might be the source of Zipf's law. The article says Benford's law (and the 1/x) results from a dimensional measurement that is scale-invariant or from the distribution of a distribution of a distribution... I do not know if word frequency is a physical measurement that is invariant under a change in scale, or if it is the distribution of a distribution of a distribution.... http://mathworld.wolfram.com/BenfordsLaw.html
So I have 3 possibilities for why language follows ~Zipf's law. My feeling is that it is not either of the above, but the 3rd possibility I mentioned before: the result of competitive positive feedback in the efficient use of symbols. The system of differential equations could cause Zipf's to fail at the upper and lower ends.
Sunday, May 8, 2016
Accuracy of Author Detection
Here is a demonstration of the accuracy of the entropy difference program in detecting authors. The bottom shows the full listing of books it was tested against, about 70 different books or collections of text by maybe 40 different authors. This method isn't the best, it's just the one I was able to easily program.
Note: the ranking average (for all the texts for the correct author) does not penalize successful detections by having a prior correct result. For example, if the correct author is spotted at rankings 1 and 2, the average correct ranking is therefore 1, not 1.5. If it ranks 1 and 3, the average rank is 1.5.
file size: 215000
AUSTIN_pride and predjudice.txt
44982 words
1 = 31525.19 = AUSTIN_sense and sensibility.txt
2 = 33416.65 = samuel-butler_THE WAY OF ALL FLESH.txt
average rank: 1
SAGAN-Cosmos part A.txt
1 = 34710.05 = SAGAN - The Cosmic Connection (1973).txt
2 = 34786.39 = SAGAN_pale_blue_dot.txt
3 = 34803.09 = SAGAN-Cosmos part B.txt
4 = 35908.95 = SAGAN - The Demon-Haunted World part A.txt
5 = 35923.25 = SAGAN - The Demon-Haunted World part B.txt
6 = 35936.53 = SAGAN The Dragons of Eden.txt
7 = 36111.48 = RIDLEY genome_autobiography_of_a_species_in_23.txt
8 = 36249 = Richard Dawkins - A Devil's Chaplain.txt
9 = 36286.77 = SAGAN - Contact.txt #### as expected, harder to detect when he changed genre
average rank: 1.29
HEINLEIN Have Space Suit.txt
1 = 36428.16 = HEINLEIN THE MOON IS A HARSH MISTRESS.txt
2 = 36771.15 = HEINLEIN Starship Troopers.txt
3 = 37019.53 = HEINLEIN Citizen of the Galaxy.txt
4 = 37223.25 = feynman_surely.txt
5 = 37377.34 = HEINLEIN Stranger in a Strange Land part A.txt
average rank: 1.25
dickens david copperfield.txt
1 = 34040.58 = dickens oliver twist part B.txt
2 = 34500.62 = dickens hard times.txt
3 = 34527.19 = dickens oliver twist part A.txt
4 = 34753.25 = dickens tale of two cities.txt
average rank: 1
twain innocents abroad part A.txt
1 = 37419.03 = twain roughing it part A.txt
2 = 37750.68 = twain4.txt
3 = 37762.04 = twain2.txt
4 = 37781.56 = twain shorts.txt
5 = 38164.64 = samuel-butler_THE WAY OF ALL FLESH.txt
6 = 38182.86 = twain - many works.txt
7 = 38192.57 = moby-dick part A.txt
8 = 38319.44 = dickens tale of two cities.txt
9 = 38375.98 = twain1.txt
average rank: 1.67
Rifkin J - The end of work.txt
1 = 1.95 = Rifkin J - The end of work.txt === oops it wasn't supposed to look at itself
2 = 32438.31 = rifkin_zero_marginal_society.txt
3 = 33556.3 = crash_proof.txt
4 = 33559.14 = brown - web of debt part B.txt
5 = 33650.69 = ridley_the_rational_optimist part B.txt
average rank: 1
RIDLEY The Red Queen part A.txt
1 = 35597.01 = RIDLEY The Red Queen part B.txt
2 = 35813.56 = Richard Dawkins - The Selfish Gene.txt
3 = 35853.03 = RIDLEY genome_autobiography_of_a_species_in_23.txt
4 = 36446.74 = Richard Dawkins - A Devil's Chaplain.txt
5 = 36564.11 = ridley_the_rational_optimist part A.txt
6 = 36670.65 = Steven-Pinker-How-the-Mind-Works.txt
7 = 36897.94 = Steven-Pinker-The-Language-Instinct.txt
8 = 36920.53 = SAGAN The Dragons of Eden.txt
9 = 36937.17 = SAGAN - The Demon-Haunted World part B.txt
10 = 36990.41 = What-Technology-Wants.txt _1.txt
11 = 37061.92 = What-Technology-Wants.txt
12 = 37061.92 = What-Technology-Wants.txt _0.txt
13 = 37115.46 = SAGAN_pale_blue_dot.txt
14 = 37124.37 = SAGAN - The Cosmic Connection (1973).txt
15 = 37197.16 = ridley_the_rational_optimist part B.txt ##### I bet he did not write this!!!!
average rank: 4.5
GREEN The Fabric of the Cosmos.txt
1 = 34597.33 = GREEN - The Elegant Universe (1999).txt
2 = 36513.55 = SAGAN_pale_blue_dot.txt
3 = 36741.75 = Richard Dawkins - A Devil's Chaplain.txt
4 = 36746.03 = SAGAN - The Demon-Haunted World part B.txt
average rank: 1
Richard Dawkins - A Devil's Chaplain.txt
1 = 35714.35 = Richard Dawkins - The Selfish Gene.txt
2 = 36146.66 = RIDLEY genome_autobiography_of_a_species_in_23.txt
3 = 36297.12 = SAGAN - The Demon-Haunted World part B.txt
4 = 36367.93 = RIDLEY The Red Queen part A.txt
average rank: 1
file size: 215000
43168 words
4 = 35905.54 = feynman_surely.txt
Note: the ranking average (for all the texts for the correct author) does not penalize successful detections by having a prior correct result. For example, if the correct author is spotted at rankings 1 and 2, the average correct ranking is therefore 1, not 1.5. If it ranks 1 and 3, the average rank is 1.5.
file size: 215000
AUSTIN_pride and predjudice.txt
44982 words
1 = 31525.19 = AUSTIN_sense and sensibility.txt
2 = 33416.65 = samuel-butler_THE WAY OF ALL FLESH.txt
average rank: 1
SAGAN-Cosmos part A.txt
1 = 34710.05 = SAGAN - The Cosmic Connection (1973).txt
2 = 34786.39 = SAGAN_pale_blue_dot.txt
3 = 34803.09 = SAGAN-Cosmos part B.txt
4 = 35908.95 = SAGAN - The Demon-Haunted World part A.txt
5 = 35923.25 = SAGAN - The Demon-Haunted World part B.txt
6 = 35936.53 = SAGAN The Dragons of Eden.txt
7 = 36111.48 = RIDLEY genome_autobiography_of_a_species_in_23.txt
8 = 36249 = Richard Dawkins - A Devil's Chaplain.txt
9 = 36286.77 = SAGAN - Contact.txt #### as expected, harder to detect when he changed genre
average rank: 1.29
HEINLEIN Have Space Suit.txt
1 = 36428.16 = HEINLEIN THE MOON IS A HARSH MISTRESS.txt
2 = 36771.15 = HEINLEIN Starship Troopers.txt
3 = 37019.53 = HEINLEIN Citizen of the Galaxy.txt
4 = 37223.25 = feynman_surely.txt
5 = 37377.34 = HEINLEIN Stranger in a Strange Land part A.txt
average rank: 1.25
dickens david copperfield.txt
1 = 34040.58 = dickens oliver twist part B.txt
2 = 34500.62 = dickens hard times.txt
3 = 34527.19 = dickens oliver twist part A.txt
4 = 34753.25 = dickens tale of two cities.txt
average rank: 1
twain innocents abroad part A.txt
1 = 37419.03 = twain roughing it part A.txt
2 = 37750.68 = twain4.txt
3 = 37762.04 = twain2.txt
4 = 37781.56 = twain shorts.txt
5 = 38164.64 = samuel-butler_THE WAY OF ALL FLESH.txt
6 = 38182.86 = twain - many works.txt
7 = 38192.57 = moby-dick part A.txt
8 = 38319.44 = dickens tale of two cities.txt
9 = 38375.98 = twain1.txt
average rank: 1.67
Rifkin J - The end of work.txt
1 = 1.95 = Rifkin J - The end of work.txt === oops it wasn't supposed to look at itself
2 = 32438.31 = rifkin_zero_marginal_society.txt
3 = 33556.3 = crash_proof.txt
4 = 33559.14 = brown - web of debt part B.txt
5 = 33650.69 = ridley_the_rational_optimist part B.txt
average rank: 1
RIDLEY The Red Queen part A.txt
1 = 35597.01 = RIDLEY The Red Queen part B.txt
2 = 35813.56 = Richard Dawkins - The Selfish Gene.txt
3 = 35853.03 = RIDLEY genome_autobiography_of_a_species_in_23.txt
4 = 36446.74 = Richard Dawkins - A Devil's Chaplain.txt
5 = 36564.11 = ridley_the_rational_optimist part A.txt
6 = 36670.65 = Steven-Pinker-How-the-Mind-Works.txt
7 = 36897.94 = Steven-Pinker-The-Language-Instinct.txt
8 = 36920.53 = SAGAN The Dragons of Eden.txt
9 = 36937.17 = SAGAN - The Demon-Haunted World part B.txt
10 = 36990.41 = What-Technology-Wants.txt _1.txt
11 = 37061.92 = What-Technology-Wants.txt
12 = 37061.92 = What-Technology-Wants.txt _0.txt
13 = 37115.46 = SAGAN_pale_blue_dot.txt
14 = 37124.37 = SAGAN - The Cosmic Connection (1973).txt
15 = 37197.16 = ridley_the_rational_optimist part B.txt ##### I bet he did not write this!!!!
average rank: 4.5
GREEN The Fabric of the Cosmos.txt
1 = 34597.33 = GREEN - The Elegant Universe (1999).txt
2 = 36513.55 = SAGAN_pale_blue_dot.txt
3 = 36741.75 = Richard Dawkins - A Devil's Chaplain.txt
4 = 36746.03 = SAGAN - The Demon-Haunted World part B.txt
average rank: 1
Richard Dawkins - A Devil's Chaplain.txt
1 = 35714.35 = Richard Dawkins - The Selfish Gene.txt
2 = 36146.66 = RIDLEY genome_autobiography_of_a_species_in_23.txt
3 = 36297.12 = SAGAN - The Demon-Haunted World part B.txt
4 = 36367.93 = RIDLEY The Red Queen part A.txt
average rank: 1
file size: 215000
satoshi_all.txt
43168 words
1 = 35144.47 = wei dai.txt
2 = 35756.13 = world_is_flat_thomas_friedman.txt
3 = 35856.63 = adam back.txt
Note: Back ranks higher here because this version of the program is trying to clean up page headings and it's deleting things out of the various author files. The public version of the program is the "official" one. It needs cleaner data files than all these books and is more accurate.
4 = 35905.54 = feynman_surely.txt
5 = 35977.79 = HEINLEIN Starship Troopers.txt
6 = 36101.18 = Richard Dawkins - A Devil's Chaplain.txt
7 = 36148.95 = What-Technology-Wants.txt _1.txt
8 = 36222.48 = Richard Dawkins - The Selfish Gene.txt
9 = 36303.8 = minsky_emotion_machines.txt
10 = 36305.12 = SAGAN - The Demon-Haunted World part B.txt
11 = 36337.96 = wander.txt
12 = 36363.81 = Steven-Pinker-How-the-Mind-Works.txt
13 = 36369.19 = SAGAN - Contact.txt
14 = 36393.73 = What-Technology-Wants.txt
15 = 36395.12 = What-Technology-Wants.txt _0.txt
16 = 36422.13 = foundation trilogy.txt
17 = 36482.69 = szabo.txt
18 = 36493.72 = Steven-Pinker-The-Language-Instinct.txt
19 = 36497.31 = SAGAN - The Demon-Haunted World part A.txt
20 = 36498.81 = SAGAN_pale_blue_dot.txt
21 = 36500.73 = Ender's Game.txt
22 = 36525.42 = HEINLEIN Citizen of the Galaxy.txt
23 = 36560.55 = RIDLEY The Red Queen part A.txt
24 = 36578.08 = craig_wright.txt
25 = 36603.95 = HEINLEIN Stranger in a Strange Land part A.txt
26 = 36614.03 = superintelligence_1.txt
27 = 36614.54 = RIDLEY genome_autobiography_of_a_species_in_23.txt
28 = 36623.71 = twain2.txt
29 = 36638.3 = GREEN The Fabric of the Cosmos.txt
30 = 36648.49 = crash_proof.txt
31 = 36693.56 = ridley_the_rational_optimist part A.txt
32 = 36698.03 = superintelligence_0.txt
33 = 36698.03 = superintelligence.txt
34 = 36706.54 = twain4.txt
35 = 36748.56 = samuel-butler_THE WAY OF ALL FLESH.txt
36 = 36777.58 = GREEN - The Elegant Universe (1999).txt
37 = 36818.65 = SAGAN - The Cosmic Connection (1973).txt
38 = 36905.35 = how to analyze people 1921 gutenberg.txt
39 = 36939.2 = twain shorts.txt
40 = 36946.28 = ridley_the_rational_optimist part B.txt
41 = 36947.92 = HEINLEIN Have Space Suit.txt
42 = 36979.58 = freud.txt
43 = 37040.28 = brown - web of debt part B.txt
44 = 37042.04 = HEINLEIN THE MOON IS A HARSH MISTRESS.txt
45 = 37060.32 = twain innocents abroad part A.txt
46 = 37089.71 = RIDLEY The Red Queen part B.txt
47 = 37097.98 = twain - many works.txt
48 = 37120.54 = SAGAN-Cosmos part B.txt
49 = 37150.83 = the social cancer - philipine core reading.txt
50 = 37166.94 = SAGAN The Dragons of Eden.txt
51 = 37176.04 = twain roughing it part A.txt
52 = 37188.02 = SAGAN-Cosmos part A.txt
53 = 37191.7 = dickens david copperfield.txt
54 = 37198.59 = The Defiant Agents - science fiction.txt
55 = 37202.43 = dickens oliver twist part B.txt
56 = 37205.45 = Catch 22.txt
57 = 37218.81 = AUSTIN_sense and sensibility.txt
58 = 37219.02 = moby-dick part A.txt
59 = 37230.43 = Justin Fox - Myth of the Rational Market2.txt
60 = 37249.28 = dickens tale of two cities.txt
61 = 37306.7 = AUSTIN_pride and predjudice.txt
62 = 37307.58 = works of edgar allen poe volume 4.txt
63 = 37309.23 = dickens hard times.txt
64 = 37320.73 = brown - web of debt part A.txt
65 = 37353.66 = moby-dick part B.txt
66 = 37408.09 = don quixote.txt
67 = 37419.12 = twain1.txt
68 = 37439.09 = rifkin_zero_marginal_society.txt
69 = 37439.73 = dickens oliver twist part A.txt
70 = 37719.14 = Rifkin J - The end of work.txt
71 = 37889.68 = J.K. Rowling Harry Potter Order of the Phoenix part A.txt
72 = 37899.77 = J.K. Rowling Harry Potter Order of the Phoenix part B.txt
73 = 37930.78 = craig wright pdfs.txt
74 = 37998.75 = Finnegans wake.txt
75 = 38169.34 = ivanhoe.txt
Subscribe to:
Posts (Atom)
next Diff = avg past N Diff * TargetInterval / Avg past N solve times.
The shorter the window average, the more protection against attacks, but there is more variation in solve times. This is unavoidable. There is a law written in stone: if difficulty is allowed to go down, you can have good protection or good solve times with a low standard deviation, but you can't have both. You have to choose how many blocks you want to "give away" by choosing the max time for say 10% of the block solves. Low block window averaging is higher protecting but wider swings in solve times. You could use N=5 for great protection if it is OK to have time to solve > 5x your target for 5% of the blocks. Once manipulators come in, you need to be prepared for 5x target 10% of the time. But such a short averaging window requires an accurate timestamp on blocks instead of miner generated times. Without that I would copy what Zcash is doing (N=17 window average with a median instead of mean for the solve times), except be sure not to use the 8% up and 16% down limits they are using, which I hope and suspect they drop before release. There is something weird with their method of getting the median that works better than the way I get the median, so us eit, which I guess comes from Digishield v3. But if you get an accurate timestamp, use the mean.
And low N averages have accidental spikes in difficulty and solve times. Miners can choose to come in immediately after those which makes the next difficulty and solve time spike even higher. so they can put it into oscillation for profit. But this might be a problem for all windows of even larger N.
The biggest protection against attacks might be to discover the methods and encourage and enable everyone to use them. That tends to block the profits of cheaters by actually leveling out the swings, helpig the constant-on miners. For example, in time warp attack is less and less useful if you initiate it and 10 people come in to take it away, splitting the profit. So maybe you shoulld give the code to enable everyone to do it. It might then become useless to everyone. Of you try to pick a bottom, but then someone comes in earlier so your bottom does not occur, and so on, until there is no bottom.
The only way I have found to get perfect protection against attackers (and fairness) and to have a perfect release schedule is to never let the difficulty drop but follow a slow steady rise, use a valid timestamp on solved blocks, and pay miners inversely proportional (Tim Olson's idea) to their solve time relative to the average time that is expected for the current difficulty setting. If a miner solves fast, he gets paid proportionally less. If he solves slow, he gets paid more. The coin release schedule stays as perfect as your clock, and there's zero profit from manipulations. The problem with a clock is that it is a third party. But it is not a problem if you're already using a subtle 3rd party going under the name of "trusted peers" who will set to a universal time clock. (The trusted timestamp also prevents timewarp attacks. ETH uses one.)
This has very important stable, real value implications. For example, miners are paid PER BLOCK for the amount of electricity needed, getting closer to the ideal of value=joules, not merely based on the average electricity expense per block expected. This requires abandoning the idea that blocks must be solved within a certain time frame. If the coin can survive post-mining on fees, then it should survive solve delays in the exact same manner to prove it can survive on fees ahead of time. But it may not result in substantial delays as everything is done so well.
This probably changes too much in bitcoin's core, and there are likely good reasons Satoshi did not do it. But it's best by starting with a known ideal and work backwards. In this case it means every time you let difficulty fall, you are harming constant-on miners relative to other types of miners.