stockpile
Stockpile is a lightweight inventory system. It provides a generic api and schema that can be used for a variety of inventory types. The Stockpile interface is separated into several layers so you can easily add, disable, or customize things like logging and caching for your application. Commonly adjusted values can be tweaked using site config variables, so you can change different settings on the fly. Stockpile supports both simple numeric count inventories as well as items properties associated with each instance in the inventory. The data is sharded by application in MySQL allowing load to be spread out over many servers.
Here's a quick example that should run right out of the box:
<?php
DB\Connection::load( array('test'=> function(){ return new DB\Driver\MySQLi( $host = '127.0.0.1', $user = NULL, $pass = NULL, $db = 'test', '3306');}));
Stockpile\Storage::attach( function ( Gaia\Stockpile\Iface $stockpile, $name ){ return 'test';} );
try {
$stockpile = new Gaia\Stockpile\Tally( $app = 'test_app', $user_id = 1);
$quantity = $stockpile->add(1000, 5 );
print "\nafter adding 5, item id 1000 has $quantity\n";
$quantity = $stockpile->subtract( 1000, 2 );
print "\nafter subtracting 2, item id 1000 has $quantity\n";
$stockpile->add(2000, 3 );
$quantity = $stockpile->get(2000);
print "\nafter adding 3, get item id 2000 returns $quantity\n";
$quantity = $stockpile->set(3000, 3);
print "\nafter setting item id 3000 to 3, the quantity is $quantity\n";
print "\nENTIRE INVENTORY\n";
print_r( $stockpile->all() );
print "\n JUST items 1000 and 2000 \n";
print_r( $stockpile->get( array(1000, 2000 ) ) );
} catch( Exception $e ){
print $e;
}
This script demonstrates all the essential features of stockpile. Questions? Read on!
Stockpile has only a handful of methods you need to know to begin using it. The first are for reading data from stockpile:
- get
- all
The get method returns a list of item quantities keyed by the ids you pass into it. Much like the memcache wrapper, if you pass in a single item id (not wrapped in an array) it returns just the value of that item, rather than in list format. The `all` method takes no arguments, but behaves exactly like you had passed a list of all of your items into `get`.
The other methods are for writing:
- add
- subtract
- set
Stockpile only manipulates one item id at a time, when writing. It is primarily an incrementing system. It adds and subtracts value. The set method really is just a convenient alias for deleting everything of a given item and then adding value back onto it.
You can find these defined in `Gaia\Stockpile\Iface`. There are a few other methods in the interface, but these are the essential ones. We will get into some of the other custom methods later on, but they are mostly for stockpile to easily do pass-through operations on wrapper classes to allow flexible customization of your stockpile object.
There are two objects at the core of Stockpile: tally, and serial. They conform to the same api and have similar goals, but store different types of data. The tally object just keeps simple tally of how many of a given item you have. The serial object keeps track of the properties of each instance of a given item. Both objects have one simple task: communicate with the database.
These two classes do all the heavy lifting of ensuring the transactional integrity of the inventories. All reads and writes of the data go through these objects. You could build an inventory out of one or the other objects here with nothing else if you wanted. The code is very simple, only a couple hundred lines, even with comments. If you need to attach individual properties to each item added to an inventory, use Stockpile\Serial as the core for your inventory. Otherwise, use Stockpile\Tally. It is much more lightweight. Stockpile\Tally keeps a simple count of how many of a given item you have in your inventory. It is very efficient at the database level.
Everything else provided in stockpile are just optional enhancements. These classes take the form of wrappers around one of these two core components. The wrappers all extend Stockpile\Passthru, which is an easy way to wrap anything that implements the Stockpile interface and pass all calls through to that inner object.
If the passthru object does nothing, all calls will be sent through to the core object untouched and return just as if the core object were the only object in play. But this wrapping allows us to observe everything going in and out of stockpile and optionally add optimizations or enhancements to behavior.
Since all of these classes implement the Stockpile\Iface, you can combine any of these classes together on the fly in flexible arrangements. Stockpile objects never keep the state of data internal to themselves. Each time you ask it a question, it returns to persistent storage to answer the question. This lowers the possibility of race conditions or having a local copy of data be out of sync with the database.
Caching is performed by Stockpile\Cacher. The cacher object wraps the core of stockpile and caches data as it reads and writes to the database.
It implements a write-thru caching scheme, which means that as data changes the cache is updated instantly without any need to expire the cache or refresh from the database. It performs row-wise caching of each item in your inventory, and maintains an index as well in the cache, to keep track of which items are in your inventory. Since the cacher inherits from the Stockpile\Passthru class it can observe all calls as they get passed inward to the core and intercept the data going in and coming out. This makes it possible for the cacher to keep the cache up-to-date and eliminate reads on to the database. As long as the cache doesn’t expire, you will never hit the database to read data.
The Stockpile\Transfer class allows you to bind the two objects in a reciprocal relationship and treat it as one object. It provides a way of keeping your balance sheets sane as you transfer items between accounts.
While you could do the work directly with the core objects, this class does most of the book-keeping for you.
The sorter object allows you to provide custom sorting of the items in your inventory:
There are some variations on this basic object that allow you to also sort by the time an item was added to your inventory.
I saved the best for last. What if you need to store a simple tally of a given item most of the time, but occasionally want to store properties for a few? Stockpile\Hybrid combines the best of both approaches by figuring out which inventory makes the most sense in each case and passing calls through to the appropriate inner tally and serial objects. There is slightly more overhead in this approach, but not too bad. And it allows you to convert tally items into serial items and back into tally when properties are no longer needed for the item. (It doesn’t do this automatically, but provides the programmer with a nice api to do so. Should it be automatic?)
This object always resides at the core of the stack, even though it wraps tally and serial. In every other respect it functions like a core object.
You can write your own wrapper to customize the Stockpile stack if you so choose. Do so at your own risk, though. It is not too difficult, but you must consider the implications of interactions with other wrappers. Really, Stockpile should give you all the functionality you need out of the box. If not, contact [email protected] to discuss.
There are two types of possible stockpile inventories:
-
tally
-
serial
Tally inventories are simple and fast. All they do is keep a count of how many of a given item you have. Serial inventories are slightly more complex. They allow you to keep track of properties associated with each individual item you add to your inventory. If every item in your inventory is always identical, you can use tally instead. If you need to differentiate between different items added to your inventory with the same item id, then you need to use serial.
Create a new tally stockpile object to access my inventory:
$inv = new Stockpile\Tally('weapon', $user_id );
The example above creates a simple stockpile object for your application, using the first parameter as an application prefix. Stockpile limits this code to a lowercase alphanumeric string (underscores are allowed too).
This object is very basic. All it does is talk to the database. In fact, it only reads and writes to one table. It doesn’t log. It doesn’t do caching. But it does everything we need it to do to read and write to our inventory. We can add other behaviors to this basic class by wrapping it with optional decorating classes.
Now let’s add caching to our basic inventory:
$inv = new Stockpile\Cacher( new Stockpile\Tally( 'weapons', $user_id ), $memcache );
The important thing to understand is that the wrapper classes behave the same as the one in the center. They just enhance the behavior, and add a few additional behaviors. In most cases, you can use the core object stockpile\tally directly without the wrappers and get the same results. The caching layer improves performance and distributes load away from the database, but it doesn’t change the core behavior of the class. The caching layer doesn’t know what is going on inside the core object. All it does is capture inputs and output finds ways to optimize performance. The Caching object only talks to memcache and to the core object we pass into it.
Instantiating serial object is similar to tally inventories:
$inv = new Stockpile\Cacher( new Stockpile\Serial( 'monster', $user_id ), $memcache );
We will get into the differences between tally and serial objects later. Instantiation is basically the same. Any wrappers that work with one will work for the other.
When constructing your application, we recommend you build a static factory method of instantiation to keep all of your objects consistent throughout your application and easy to manage. Let’s say i want to build an animalfarm with caching, using just a simple tally count. Here is how i might set it up:
class AnimalFarm {
public static function inventory( $user_id ){
return new Stockpile\Cacher( new Stockpile\Tally( 'animalfarm', $user_id ), $cacher );
}
}
Now i can use my object reliably using a standard interface:
$inv = AnimalFarm::inventory( $user_id );
This gives me an instantiation pattern throughout my application that I can change as needed. When first prototyping the application, maybe the static factory method only returns the Stockpile\Tally object with no wrappers. This keeps everything simple and straightforward. If something doesn’t work, you only have to look in one or two spots. As you add functionality to your application, you can add the other elements.
Stockpile uses the DB\Transaction objects. You can wrap your entire operation in a transaction by doing:
Transaction::start();
$stockpile->add($item1, 1);
$stockpile->subtract($item2, 1);
Transaction::commit();
If you don't start a transaction, stockpile will start and commit its own transactions internally for each operation where necessary.
Before going further we should discuss stockpile quantity. Most operations in Stockpile return some sort of quantity. When you add or subtract, it returns the new quantity. When you fetch an item, you get the quantity of that item. The tally and serial objects have different ways of representing quantity. With tally, the quantity returned can be described with a simple integer. With Stockpile\Serial, the quantity of an item is described by all the serials attached to a particular item id and all the properties. We encapsulate this quantity with a Stockpile\Quantity object.
Stockpile\Quantity has several methods you can use:
- value
- serials
- all
- get
- set
If you just want to treat the quantity as a scalar value, you can do so:
$quantity = Stockpile\Base::quantify( $inv->get( $item_id ) );
That is the same as doing:
$quantity = $inv->get( $item_id )->value();
Note: because php is stupid, inval( $quantity ) doesn’t do what you’d expect. We have to be careful with intval anyway in php because it can cause problems with bigint. Better to force it to a string or just let Stockpile represent it for you. This is less of a concern in most cases since when writing application code, you know which type of inventory you are dealing with. We will get into how to manipulate the quantity object in some examples below. The main thing to realize is that the quantity object is a list of serials with properties associated with each one:
foreach( $quantity->all() as $serial => $properties ){ }
The properties will always be an array, but other than that, will be free-form values supplied by your application.
Read back the quantity of an item in my inventory:
$quantity = $inv->get( $item_id );
If the item isn’t found, it returns an empty record. With a tally inventory, this is the number zero. Stockpile\Serial will return a Stockpile\Quantity object as usual, but with a count of zero.
Multi-gets for items are a snap as well:
foreach( $inv->get( $item_ids ) as $item_id => $quantity ) { /* do work here */ }
The returned data set is a simple key value pairing of item_id/quantity. Only those items found in the inventory will be returned. Zero quantity items will not appear in this list. Not to beat a dead horse, but tally objects return integer quantities and serial objects return Stockpile\Quantity integer objects.
The method for reading all items in your inventory is very similar to the multi-get call:
foreach( $inv->all() as $item_id => $quantity ) { /* do work here */ }
We don’t provide any real searching or sorting beyond this. More on why later.
To add an item to your inventory just do:
$new_quantity = $inv->add( $item_id );
This will add a single item to your inventory. If that item already exists, it will increment the value for that item. The method always returns the new value of your inventory after the change has been applied. So, if I had 10 of that item_id already in my inventory, the method would return 11.
To remove an item from your inventory:
$new_quantity = $inv->subtract( $item_id );
The interface for subtracting is itentical to the one for adding. It just reverses the direction. Stockpile will throw an exception if you do not have enough of the item to perform the subtraction, and roll back any attached transactions. Again, the method returns the amount for that item that exists in your inventory. If you are performing this operation within a transaction, it returns the number that will exist in your inventory at the point the transaction is committed.
What if you want to add 10 of a given item to my inventory?
$new_quantity = $inv->add( $item_id, 10 );
Simple as that.
Subtraction method behaves the same way:
$new_quantity = $inv->subtract( $item_id, 10 );
So far, the examples given are for the Tally type of inventory. Serials are not that different from tally on the surface. Instantiation is the same. You can add and subtract items the same way you would with the tally inventories. Underneath, the data structure is very different. Instead of 1 row for each item you have, there are many rows for each item, with a serial number assigned to each row. In the serial api, if i have 5 of a given item, there will be 5 entries in the serial table, one for each serial.
Stockpile tries to abstract this complexity away from you while still giving you access to the hooks if you need it:
$new_quantity = $inv->add( $item_id, 5 );
When this is called, Stockpile converts the integer passed in into a quantity object. The quantity object knows that you will need 5 new serial numbers, so it generates them by calling the counter dao and populates these serials into an internal lookup table along with placeholder properties entries. The properties are empty by default, but if you create your own custom Stockpile\Quantity object, you can have these variables populated with custom default values. More on that later.
Let’s pass the quantity object in directly, and actually assign some properties.
$quantity =$inv->quantity();
$quantity->add( array('xp'=>22, 'health'=>5) );
$quantity->add( array('xp'=>39, 'health'=>2) );
$quantity->add( array('xp'=>1, 'health'=>1) );
$result = $inv->add( $item_id, $quantity );
When you call the $quantity->add() method, the serials are generated on the fly. These serial numbers are unique across all of stockpile and are never used again.
Once when we add the quantity object to our serial inventory, these serials are populated into rows in the db, along with the properties we gave them.
If you already have a bunch of serial numbers, you can add them via:
$serials = array(1,2,3,4);
$quantity = $inv->add( $item_id, $serials );
Serials, when generated, are unique across the entire system, but if you supply your own, we assume you obtained those serial id numbers from a unique id generator that will be unique in stockpile. Stockpile doesn’t really care and only enforces that the serial is unique for that user/item_id pairing.
We can delete items in the same way as we added them, manipulating the inventory the same way we would Stockpile\Tally:
$inv->subtract( $item_id, 2 );
Stockpile assumes you don’t care which two are deleted and just pops off a couple from the inventory and throws them away. But what if you do care? If you want to subtract specific serial rows from the inventory, you need to pass those values in. Enter the Stockpile\Quantity object! You can use it to tell stockpile which specific two serial rows to remove:
$inv->subtract( $item_id, $inv->quantity( array( 1000113345, 1121211222 ) ) );
You can also just pass in the list of serials you want to remove, like this:
$inv->subtract( $item_id, array( 1000113345, 1121211222 ) );
Stockpile figures out what you mean and does the right thing.
So far we haven’t discussed using the set( $item_id, $quantity ) method. And with good reason. Inventory systems should be adding and subtracting from your totals, not forcing it to a given amount. Adding and subtracting approaches avoid race conditions that could clobber data and result in loss or duplication. However, sometimes when writing an admin panel, you may need to force a user’s inventory to a given quantity. The set method does this for you, but maybe not as you might think. It is really just a nice wrapper around add/subtract, where it calculates the difference between where the inventory currently is and the desired amount and then performs add or subtract as needed.
Using what I described above you can probably write your own mechanisms for trading or marketplace transfers. But as a way to make it easier I provide a nice interface to do that. Let’s re-use our AnimalFarm example from earlier:
DB\Transaction::start();
$trade = new Stockpile\Transfer( AnimalFarm::inventory($user_id ), AnimalFarm::inventory($other_id ) );
$trade->subtract( $bird, 5);
$trade->add( $sheep, 2 );
DB\Transaction::commit();
This example takes 5 birds from my inventory, and puts them in the other person’s inventory. And it takes 2 sheep and puts them in mine. Since I wrapped it in a transaction, it performs it in an atomic operation that commits all of the changes at once. If either party doesn’t have enough to cover the trade, Stockpile throws an exception.
The nice thing about the Stockpile\Transfer object is that it is a wrapper for your regular inventory object. You can use it exactly as you would use the underlying inventory object. All the reads will behave the same. But when you do writes, it uses the other account as the provider for adds, and the recipient of the amounts subtracted. Make sense?
You can transfer items between users, and between applications. It has the nice side-effect of creating the building blocks for an escrow service. This is useful for creating marketplace or trading infrastructure. I can transfer an item from my main application to another application that is not normally accessible to the user until we are ready to send it on its way. In case all that fancy wrapper stuff confused you in the previous example, here is a bare bones example with no additional factory methods.
DB\Transaction::start();
$inv = new StockPile\Tally('animal', $user_id );
$escrow = StockPile\Tally('animal_escrow', $user_id );
$inv = new Stockpile\Transfer( $inv, $escrow );
$inv->subtract( $sheep, 1);
DB\Transaction::commit();
This example moves 1 sheep item out of my inventory and places it in the animal_escrow application. I am just using the animal_escrow name by convention. There is nothing special about it. Later, I can move the the item to the user who purchased it and transfer gold from their account into my own. I assume in this example that gold is an item id just like anything else and can exist inside the animalfarm inventory (We’ll look at a more complex example in a few minutes).
Another note about instantiation ... if you don’t use the caching or logging wrappers in one spot, don’t use them anywhere else in your application either, or you might see inconsistent results. If you use caching, make sure your application uses the Stockpile\Cacher wrapper everywhere. Otherwise you could create inconsistencies in the cache for your users and stale data. That is why I recommend setting up the factory methods.
Here is another example of escrowing:
// start a transaction for this operation
DB\Transaction::start();
// set up the escrow account where the item is being held on the marketplace.
$owner_market = new Stockpile\Tally('animal_market', $owner_id );
// set up the main owner account as well
$owner = new Stockpile\Tally('animal_farm', $owner_id );
// set up the main account of the purchaser who is buying the animal.
$purchaser = new Stockpile\Tally('animal_farm', $purchaser_id );
// wrap the owner in a transfer so we can take the gold from the purchaser
$owner = new Stockpile\Transfer($owner, $purchaser);
// wrap the purchaser in a transfer so we can move the escrowed item into their account
$purchaser = new Stockpile\Transfer( $purchaser, $owner_market );
// take the gold from the purchaser
$owner->add( $gold, 1000);
// take the sheep from the owner's escrow account on the marketplace.
$purchaser->add( $sheep, 1);
// commit the transaction
DB\Transaction::commit();
This should transfer the item out of the owner’s escrow into the purchaser’s account and take gold from the purchaser and transfer it to the owner. If anyone doesn’t have enough to cover the cost, an exception is thrown and the transaction is rolled back.
In the example above, gold, animals, and all other items were stored in the same inventory, each with a unique item id. But what if you have a system where the gold is in one of a user’s accounts, and the animals are in another. Let’s consider how that example might look:
DB\Transaction::start();
$owner = new Stockpile\Transfer( new Stockpile\Tally('animal_currency', $owner_id ), new Stockpile\Tally('animal_currency', $purchaser_id ) );
$purchaser = new Stockpile\Transfer( Stockpile\Tally('animal_farm', $purchaser_id ), new Stockpile\Tally('animal_market', $owner_id ) );
$owner->add( $gold, 1000);
$purchaser->add( $sheep, 1);
DB\Transaction::commit();
To transfer serials from one person’s inventory to another, stockpile needs to know which serials and properties you are transferring. If you pass in just an amount, stockpile will try to figure it out for you and transfer the correct amount, picking the more recent serials from the list. However, you probably should specify the serial numbers and properties explicitly:
DB\Transaction::start();
$me = new Stockpile\Serial( 'animalfarm', $user_id );
$other = new Stockpile\Serial( 'animalfarm', $other_id );
$transfer = new Stockpile\Transfer( $me, $other );
$quantity = $me->get($item_id)->grab( $serials );
$transfer->add( $item_id, $quantity );
DB\Transaction::commit();
The grab function targets specific serial rows in a quantity object and encapsulates them in a new serial object. Then we can pass it through to the transfer object.
Stockpile uses row-wise caching to keep an accurate picture of your inventory always ready in the cache. As data changes, the cache is overwritten instantly, even before any transactions have been committed. For more detail, read the section below on caching.
Caching is turned on in the default Stockpile object. You can bypass it easily enough by creating your own custom instantiation:
$inv = new Stockpile\Cacher( new Stockpile\Tally( $app, $user_id ), $memcache );
Sometimes, to verify something during the development or debugging, you may want to force the cache to be refreshed. Stockpile\Cacher has a hook for that:
$stockpile->forceRefresh( TRUE );
Any subsequent calls to that object will cause any cached values to be read from persistent storage and overwritten into the cache.
Stockpile has one hook that resolves a given account in a stockpile inventory to a DB\Connection name. You attach a callback handler to the storage object to define a rule for which connection to connect to.
Stockpile\Storage::attach(
function ( Stockpile\Iface $stockpile, $name ){
return 'mydb';
}
);
This particular handler would always tell stockpile to use Connection::instance('mydb') no matter what application or user. But you could be much more creative. You could use the Shard\VBucket method to handle it and shard based on application name prefix and shard of the user id.
Stockpile\Storage::attach(
function ( Gaia\Stockpile\Iface $stockpile, $name ){
$vbucket = new Shard\VBucket( range(1,5) );
return $name . $vbucket->shard($stockpile->user());
}
);
The `Stockpile\Serial` object allows you to attach a callback handler to validate the quantity objects that come in and out. This allows you to sanitize variables and clean them up as they come out of the cache, or the database and make sure they conform to your expectations. Can even use it to populate default values. First, set up your validation function:
class AnimalFarm {
public static function validate( Stockpile\Quantity $q ){
foreach( $q->all() as $serial => $properties ){
if( isset( $properties['xp'] ) ) continue;
$properties['xp'] = '1';
$q->set( $serial, $properties );
}
}
}
Now you can use it to make sure that the xp property is always set and defaults to 1:
$stockpile = new Stockpile\Serial( $app, $user_id );
$stockpile = new Stockpile\QuantityInspector( $stockpile, array('AnimalFarm', 'validate') );
If you do this step in your factory method of instantiation, everywhere the stockpile object is used will be sure to use this validation routine.
In order to keep the api clean and efficient, Stockpile simply tracks user_id, item_id, and quantity. By default it does not support sorting. The best solution is probably providing those kinds of hooks at the item definition level where users can all share in the common groupings or sort orders you have defined at that level. Then pass in those item ids to Stockpile to retrieve the item quantity information for your users’ inventories.
However, if you really need sorting of items in stockpile, there are a few classes I provide to help you do that.
- Stockpile\Sorter
- Stockpile\RecentSorter
- Stockpile\FirstAddedSorter
The classes store sorting information in a separate table and use the passthru interface to add hooks in where necessary to keep track of sorting information for stockpile. The RecentSorter class keeps track of when you add new stuff to stockpile and moves them to the front. You can use it as any other wrapper:
$stockpile = new Stockpile\RecentSorter( new Stockpile\Cacher( new Stockpile\Tally( $app, $user_id ), $memcache ) );
Be sure to wrap the sorter objects around the cacher object since the sorter objects need to sort the items after they come out of the cache. Every time you call $stockpile->add($item\id), stockpile moves that item to the top of the list. When you do $stockpile->all() or $stockpile->get( $item_ids ), the sorter object will use the defined sorting mechanism defined to sort your result and return it back to you.
The default Stockpile\Sorter object doesn’t store any sorting information by default. It assumes you will tell it the order, by calling: $stockpile->sort( $item_ids ); stockpile stores the sort order in the same order you have sorted your item ids. then later, those items will always show up at the top of the list. Since RecentSorter extends Stockpile\Sorter, you get this functionality for free in that class. Both can be used in conjunction with each other. the Stockpile\Sorter::sort function uses integer values way above the range of the values used by the recentsorter.
I am still not quite sure how well all of this will work in practice, so use at your own risk. Send an email to [email protected] if you want to brainstorm on how to make this most effective.
The code replies on SQL math to verify there is enough of an amount in the account to subtract an item. Since queries are processed in serial order, this should be repeatable and predictable. Only time this becomes scary is during dsn swap if the slave runs behind in replication at all. Any other cases you can think of? My application applies the update and if it gets affected rows < 1 it knows there wasn’t enough, and throws an exception after reverting the transaction.
As data changes, the changes are written to the cache instantly, ahead of transaction commit. This is safer than it sounds, for severa; reasons. We never rely on the cache as the basis for updating the database or making any transactional decisions. All updates to the cache are gotten from the database using some SQL trickery. If a transaction is rolled back after the data is written to the cache, a callback hook deletes all the changed cache keys automatically, bringing us back in line with the db. While there is a very narrow possibility that a user might encounter a race condition that could cause the cache to be overwritten by a staler version of the data, this will never affect the long-term integrity of the persistent data.
For the cache to be populated with stale data the following condition would need to occur: A transaction would need to change a row, then update the cache, then hit a fatal error before committing, preventing the callback from deleting the cache, or a network error prevents it from deleting the cache. In this case, the cache will be stale. I can’t think of a case where a true race condition would occur that would clobber the data. Since the db holds a row lock on the row we are changing, the second write to the cache cannot be out of sequence since the open transaction holds the row lock and has already written to the cache.
The risks of incorrect or stale data in the cache with this design are as low or lower than other caching mechanisms we’ve used. we always get the correct value for the cache from the db via an mysql variable select from the insert/update operation.
Other race condition happens when the data is missing from the cache ... but in that case we do a memcache add so that if the data is updated at the precise moment an update occurs, our cache re-populate operation doesn’t clobber newer data from the update.