-
-
Notifications
You must be signed in to change notification settings - Fork 322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Balance assertions that check only cleared postings #556
Comments
Interesting! Thoughts:
|
Hmm, I guess if you only use balance assertions with accounts that are all-cleared (like cash or things imported from CSV), then you can't hit the issue. As for making the syntax something ignored by standard hledger, then I guess this would mean not using hledger assertions but some special-form comments that would be parsed only by the add-on command? That would lose the ability to be informed of an assertion breakage by just running As for my workflow, my ideal workflow would be:
My current workflow, imposed by balance assertions applying also to uncleared postings, is a change in step 2, which becomes adding the following transaction to my journal:
Which has the unfortunate side effect of pushing all pending credit card purchases to the end of Hoping my workflow is now clearer, |
I think this feature needs further discussion/experimentation. |
That was a bit unclear. I meant: sorry but with available brainpower I still don't fully understand the need for this, or the proposed solution. If you want to move this forward, the best way would be to (a) try to write the documentation for it (b) try to implement it yourself, asking for help as needed (c) use it and flush out issues and/or (d) raise it on the mail list and try to build interest. |
I have a similar need to what @Ekleog has explained above, but I think for a slightly different reason. I like to leave balance assertions in my documents so that i can refactor without fear, knowing that nothing changed. I add balance assertions for my bank accounts and credit cards most times I enter transactions. The most common time that I would run into the need of "assert cleared" would be something like this:
I use Ledger now, and deal with this situation by breaking the "Reconcile" transaction into two, and ordering the transactions so that the parse order puts the Bank reconcile before the credit card payment, and the CreditCard reconcile after the payment. I can't do this and also keep the data accurate in HLedger, since it sorts by date first. The system proposed by @Ekleog in the top comment would be much preferable to me, and I'd probably switch to HLedger just for that feature. |
I'll need a reproducible example to understand.. |
With the example from my previous message (which I have slightly modified to use the correct syntax for comments), I get the below problem. Given the situation explained in the file, I don't see a way to keep these balance assertions and make them pass, without making the dates of transactions incorrect. The proposal in this issue would allow me to change the balance assertion to $ hledger -f ordering.journal balance
hledger: balance assertion error in "/home/mythmon/tmp/ordering.journal" (lines 22-26):
in transaction:
2018/01/05 * Reconcile
Assets:Bank 0 = $0
; this assertion would currently fail, even though it
; accurately reflects the real world
Liabilities:CreditCard 0 = $1
after posting:
Assets:Bank 0
balance assertion details:
date: 2018/01/05
account: Assets:Bank
commodity: $
calculated: $-2
asserted: 0 (difference: +$2) |
Thanks @mythmon. I don't see how your assertions should be expected to pass. Here's what I and hledger expect after the above txns:
|
The important part is that the credit card payment transaction is half cleared: the credit card split is cleared, but the bank split is not yet present on the bank's website. I want a way to make a balance assertion that happens on a future date that accounts for this. The balance assertion in my file reflects what the bank and credit card company tell me on my statements. Since those statements are out of sync, there isn't a way to model the situation with hledger's current feature set. |
I get generally what you're saying. Aren't there some practical workarounds, given the existing feature set ? Beyond that, the next step is to draft user documentation for the feature, and then discuss more widely if the benefit outweighs costs. |
I'm not sure if there is a practical work around. Like I said, I have a work around that works in Ledger, but since I can't control the order of transactions on different dates in HLedger, that doesn't work here. |
The workaround I currently use on |
I don't think that changing the data to be incorrect is a good workaround. |
My current practice is to put |
Hmm, do you have an example of how you do it? My current understanding of your usage is something like:
Which |
@Ekleog: The above parses for me (hledger-1.9). The balance assertion fails of course, or passes if you change the asserted amount to -3. This is as expected. What I would do, if I wanted to be precise:
or more briefly:
|
Yes, I meant that the balance assertion fails, indeed. The issue with the solution you put forward is that you need to wait the clearance date in order to enable the balance assertion. And, here, on the 5th of the month (approx., if only it was that precise), all transactions up to the 31st of previous month are cleared. Which means that there is never a clean “all has been cleared” state. And thus there is no point in time when a balance assertion could cleanly state “this is the amount of money in the account”, it would necessarily have to include not-yet-cleared data. (BTW, I use separate postings for balance assertions, as I don't have balance assertions on each posting, I'm not really sure how that changes behaviour wrt. what you proposed and having many transactions cleared on the same date) This is why, currently, my solution is to use |
Sorry for adding confusion. Instead of "leave the balance assertion disabled", I should have said "leave the clearance date unspecified". Or "leave a note to adjust this posting date later". Sometimes indeed I might disable the balance assertion as well, by commenting it out: Back to your issue: as is often the case, there are many implicit details in our individual setups and workflows that are hard to communicate and understand completely. I'm certain what you're saying makes sense in your situation, but I am not able to see clearly the real problem right now. I could comment on a specific proposal to improve hledger, or perhaps someone else can suggest another way you could handle it with the existing feature set. |
For my workflow, it is basically “I add transactions when they are performed, and when I receive a bank report, I add in an assertion with a However, with credit card, there are pending transactions, even when the bank report is produced, and which don't appear on the bank record. For this reason, I'd like to have a way to assert an amount for an account, counting only cleared transactions. Hence this idea, from the first post, to have an Hopefully it is more clear now? :) |
Nope. :) I'm sorry, my brain won't take this in. |
More clear after exercise. My workflow is different so I'm not the best person to judge how worthwhile this is, but I welcome experiments. Be aware that we have proposed similar extensions to the assertion syntax for another purpose (controlling whether subaccounts and other commodies are included). |
PS it could be interesting to share the above on the mail lists, especially Ledger's, and see how other folks have solved this. |
Given the feature as you describe it, what would you do once those transactions clear ? I foresee complications. |
What I do when the transactions clear is add the For the syntax, the proposed syntax for subaccounts is the same as the proposed syntax here, indeed… maybe something like this would look good?
It could even generalize to
for an account with 1234 expected once everything clears, 5678 counting only pending (and cleared) transactions, and 9012 counting only cleared transactions. Does that sound like a not-too-bad syntax to you? Will ping the ledger ML as soon as I get some time to write a wrap-up mail of this conversation :) |
I parse statements from both bank and credit card company with some
scripts, and since both statements include "balance (after this
transaction)" column, I parse it as well and convert to balance assertion
for every transaction parsed (via "balance" csv import field.
I have assets:bank and "liabilities:card company:my card". During month,
'liabilities:card company:my card" slowly goes into red. Then statement
arrives and is parsed. Statement will contain "payment received" line,
which would be imported as transaction from "liabilities:card
company:payments" to "liabilities:card company:my card".
Whenever bank statement is parsed, any payments to credi card company are
recorded as going from "assets:bank" to "liabilities:card company:payments".
This way, "liabilities:card company:payments" will at any point contain
"uncleared" amount of money, which ideally should be zero, but due to time
differences in described by other posters earlier, this account is never
quite flat. It does allow me to quickly see how much money is unreconciled
between bank and credit card.
I do not bother marking transactions as cleared explicitly.
…On Sun, Jun 17, 2018 at 5:34 PM, Léo Gaspard ***@***.***> wrote:
What I do when the transactions clear is add the * and a ; date: with the
date at which the transaction cleared, which is after the assertion, so
everything should work. :)
For the syntax, the proposed syntax for subaccounts is the same as the
proposed syntax here, indeed… maybe something like this would look good?
2000-01-01 * balance assertion
account 0 = *:1234
It could even generalize to
2000-01-01 * balance assertion
account 0 = 1234 !:5678 *:9012
for an account with 1234 expected once everything clears, 5678 counting
only pending (and cleared) transactions, and 9012 counting only cleared
transactions.
Does that sound like a not-too-bad syntax to you?
Will ping the ledger ML as soon as I get some time to write a wrap-up mail
of this conversation :)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#556 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHNKrkhx32t5WuxnkYq_BPvUe2dZCkkks5t9oUhgaJpZM4Nl2Y->
.
--
D. Astapov
|
@Ekleog let us know if you get a chance to try adept's method (using a temporary third account) . |
I've already considered it, unfortunately that means not tracking transactions individually, and thus not knowing from when the error came, if there ever is an error, so rejected the option :/ (still have somewhere on my todo-list to ping the ML about this, sorry for not having done it yet, I'm a bit overwhelmed currently) |
No hurry! Thanks for the update |
Leo: assuming that you can parse statements from your bank/credit card
company, and have balance assertions on every transaction, what kind of
error are you worried about?
…On Mon, Jul 2, 2018 at 1:02 PM, Simon Michael ***@***.***> wrote:
No hurry! Thanks for the update
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#556 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHNKreuh852UFloujwsM1niGjNSCihxks5uCgvOgaJpZM4Nl2Y->
.
--
D. Astapov
|
On 07/02/2018 09:19 PM, Dmitry Astapov wrote:
Leo: assuming that you can parse statements from your bank/credit card
company, and have balance assertions on every transaction, what kind of
error are you worried about?
Well, I'm worried about the bank not counting a transaction, or me
ill-counting it. Like:
* I write down $10 from liabilities:inflight to expenses:*
* Actually, it was $5
* When I add in a parsed statement, $5 go from assets:credit-card to
liabilities:inflight
And now liabilities:inflight will forever be in the red by $5. Until I
add in a balance assertion and I have to figure out where the error was.
As for adding a balance assertion on every transaction, I must say I
can't see how it'd help, as that's a mismatch between manually-inputted
and automatically-inputted data? (also, anyway I'm usually using the
Android app My Expenses for manual logging, and then export and convert
it to ledger format, so I couldn't easily add in balance assertions)
|
I have a similar workflow/issue: while I do try to add transactions as they occur, my primary sources are bank statements, pay stubs, that sort of thing. Particularly, I have a portion of my salary automatically transferred to a retirement fund, so that account is involved in income transactions every couple weeks. However, those are then deferred for up to a couple months, before a number of them are all added at once. Knowing that, when I enter my pay stubs, I mark the retirement postings as "pending" until they show up in one of the statements for that account. Meanwhile, when I get a statement there, I mark the postings it includes as "completed" with the later date (but note that there will always be some later ones still pending), record interest and other activity within the account, and add a balance assertation to the very last posting involved, just to be sure I copied everything correctly and that no external file adds a transaction throwing everything off. However, by definition, the balance on the statement, and therefore the assertation, doesn't include pending activity. For example, this last august, the account register lists four salary deferments (mostly from june) coming in on the sixth; interest being added on the sixteenth; a few investment stocks bought on the twentieth; and a pair of pending postings currently dated the third and the seventeenth according to when the pay stub was dated, but which will not arrive in the account until probably october, though may perhaps be included next month. As the investments are the last completed posting, the period-end balance should be asserted there, but as the two pending postings are still listed on their original disbursement days, I have to comment the assertation out for another month or two until I recieve the statement including them. That doesn't reflect the true balance of my account; I don't have nearly one hundred extra dollars there just because the postings didn't appear on last month's statement, and if this were an expense account, I wouldn't be able to draw from them to cover anything. Adding a "valid" balance assertation would require involving transactions that in actuality are only incidentally related, doing math that ends up disagreeing with the bank's, and then undoing it all once the statement comes out that does list those postings. Avoiding that "only" requires putting off being able to verify the balance, and remembering to check old assertations to see if they can be un-commented. I'm sure there are people whose workflow does involve including those pending transactions in balances, so while doing so makes little sense to me, I agree with the proposal @Ekleog put forth to use new syntax for excluding them. However, while it's certainly not anything I'll stop using hledger over, the current restricted behaviour does definitely add unnecessary hassle to managing my accounts. |
Thanks for the writeup. Have you tried @adept's suggestion, moving the funds into a temporary account ? That is an accurate modelling of the situation using the tools we have, I believe. |
Léo Gaspard: sorry for necroposting, I managed to miss your email. I ran
into issue you described once or twice, but In my experience, doing
'hledger register' on transfer account allowed be to spot problem pretty
quicky.
Let's say that I have credit card account with lots of small puchases that
I parse from CSV, and that CSV will include "balance paid" transactions
from time to time, which i would import as going from "transfer" account. I
also have bank account that periodically replenishes credit card account by
sending money to transfer account - also auto-parsed from CSV. I oftern pay
off balance manyally, but usually it is just one transaction from bank per
month. Then register of transfer account will look roughly like this:
[money in from bank]
[money out to credit card]
[money in from bank]
[money out to credit card]
[money in from bank]
[money out to credit card]
[money in from bank] ; manual payment
[money in from bank] ; manual payment
[money in from bank]
[money out to credit card]
Imagine that there was some disconnect and at the moment transfer account
is $5 in credit that I cant explain. I will print out register for transfer
account and as I scroll through it, there would be many many points at
which transfer account is flat. I need to find the last time it was flat
and basically bisect between that point and the end of register - error
should be there somewhere. In my experience it is usually pretty easy to
spot.
Obviously, your experience could be different and if so I would like to
hear about it.
…On Fri, Sep 14, 2018 at 5:06 AM Simon Michael ***@***.***> wrote:
Thanks for the writeup. Have you tried @adept <https://github.com/adept>'s
suggestion, moving the funds into a temporary account ?
That is an accurate modelling of the situation using the tools we have, I
believe.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#556 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHNKiNdDDaOKhOYORl3X9pXwUwGPfscks5uays5gaJpZM4Nl2Y->
.
--
D. Astapov
|
@simonmichael If I get tired of the comment shenanigans, it does seem like a viable workaround, but for now it's not a particularly large hassle. I mainly wanted to add my voice in support and provide a(nother) concrete, real-world example. I have pulled the source and think I might be able to pick apart the logic if we do wind up wanting to add this (the biggest difficulty is likely that I don't otherwise use Stack), but we'd need to figure out the syntax first:
Another thing we'd need to keep in mind is that both the second and third may break the library api (more likely with the third, but the second depends on how posting dates are already implemented), and the second has the possibility of breaking people's existing files (though a good choice of tag name could severely reduce the number of people already using it for something else). Thoughts, anyone? |
There shouldn't be any need to use stack if you don't want to, aside from that's what we document. Being familiar with the implementation and interactions of the current assertions functionality, I'm very cautious about adding more complexity to assertions. If you do try this, I think the most economic way would be to draft some mockups of the new docs and small examples of input and expected output. We can change the api when needed. Breaking existing journal files is to be avoided if possible, indeed. |
Hmm, I'm not sure I understand your scheme, then. My understanding of it was to do the following, assuming an expense of 1 on 2000-01-10 and an expense of 2 on 2000-01-15:
My problem with my bank is that the expenses for 2000-01-01 to 2000-01-31 are all withdrawn around 2000-02-05 ; which means there is often zero times when the transfer account is at 0. Which makes mistakes hard to notice. Is my problem more clear now? Did I misunderstand your proposed solution? @ag-eitilt Great to know you're working on this! Thank you 👍 Also leaning for the third option, FWIW. |
@Ekleog in my example you would rather have something like this (assuming that you are actually spending from account called .... erm ... lets call it "whatever account name you want"):
So on 2000-02-05 you transfer account goes flat and stays flat until the end of next month. |
@adept Oh I think I understand, you actually use two transfer accounts so that one goes flat every month from -05 to -31 and the other (let's say That sounds like it'd work, indeed. Currently my workaround has been surrounding hledger with a script to automatically add Thank you for your idea! That said I must add that an actual way to encode these still sounds useful to me, as it would better encode which transactions should be taken into account when transactions have varying dates -- case in mind, checks. Checks are just as complex but without the regularity of credit card. Even though I don't do these really often they've also proved to be a PITA to handle with assertions -- much more than credit card (and I don't think either of our solutions would apply to checks), but given I use them so little I didn't think of mentioning them initially :) |
I wouldn't call "whatever account name you want" a "second transfer account". From your descriptions it is a real account (i get that it is not a credit card, but by the sounds of it it is something similar). The only transfer account there is "transfer" |
Separate from the discussion in progress, which is a bit too short on examples for me to follow, here's another response to @Ekleog's original example. Here it is with the posting date added to make all assertions pass:
When I find myself in this situation, I'll either
Is this so unpleasant ? Enough to justify adding a new feature ? You say the posting date "pollutes the register output", but I feel it simply makes the register more accurate. |
@simonmichael Your third option is exactly what I've been doing. The problem is when you have a few such transactions per day, it makes an awful lot of TODOs to come back to later. (the “not bother with that particular assertion” is unfortunately not a solution, with credit card accounts here there are always “inflight” transactions, so that would mean never asserting amounts) |
@Ekleog so my reading is that you do not have 'credit card' account in your account plan, you are basically keeping credit card transactions in account that you also use for other things (my guess would be it is you 'checking' or 'current' bank account). If I am right about this, what is your reasoning behind this decision? |
@adept You are right indeed. My reasoning is it's the line my bank gives me, which makes for easy assertions: just copy the number from the bank and have hledger assert there's likely no error (or the bank's error and mine are of equal magnitudes). Except the bank's value somehow lags behind hledger's computed value, because it only counts |
@Ekleog I think that you are really making it hard on yourself by not using a separate account, and if you had bank:checking and bank:credit accounts, you would've had an easier time reconciling them. Bot obviously everyone does their own thing with finances. |
@adept TBH, I wouldn't know what to put in the separate account. I really have a single amount of money from which I can withdraw by using credit card, ATMs, checks, going to the bank counter, etc. The withdrawals using credit card or checks are just delayed until later (which would ideally be the time where I mark them |
@Ekleog so you are recording transactions manually as they happen, and then later on you get a statement from bank and manually mark them as cleared? Or is there some degree of automation involved (like CSV imports)? |
@adept Yes. No automation yet (well, apart from the fact I record my transactions on MyExpenses then export from phone to hledger), but someday I'll get to writing some script to automatically match uncleared transactions with bank transactions that have the same day&amount and just ask for validation :) Currently my workaround works at the “import from MyExpenses” phase, where I add in a But this scheme still doesn't work with checks, which have been the most painful thing to handle with assertions to date :/ maybe I should just consider a signed not-cashed check as a liability, but… technically it's not a liability, it's a transaction that has just not yet been cleared, so it'd be great if I could actually encode it as it is, and not in a way contorted to please the technical restrictions :) |
@Ekleog Just a thought: If you have a way to get CSV from your bank, then you can manually record transactions in "bank:cheques" and "bank:cc" and so and and then import CSV from bank with account1="bank" and account2 = "bank:cc" or "bank"cheques" according to import rules, and then balance of bank subaccounts will reflect the uncleared balance at all times. Sorry for highjacking the discussion, I will probably bow out at this point. |
This is a feature idea that I think may be interesting for easier reconciliation: having balance assertions that match only on the sum of cleared postings.
This way, files like this one
would pass balance assertions without having to fiddle with transaction 1's posting 2's date (which would pollute register output).
(obviously, exact syntax is TBD, I would have loved to just use whether the posting on which the assertion is made is cleared or not, but that would break existing ledgers, so a separate syntax has to be picked)
What do you think about it?
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
The text was updated successfully, but these errors were encountered: