Missing Data

[JeremyWhittaker]

I'm pulling financial data for symbol LEN. My data frame is below as well as a visual chart for revenues. I'm missing chunks of data. For small companies, this may be normal as perhaps they didn't release updated quarterly financials etc. But this company did in fact release this data. I was able to verify by looking at Yahoo Finance(screenshot also below). Is there going to be a lot of companies where your data isn't accurate like this? When I'm trying to analyze aggregate data this makes it incredibly difficult.

The python function I used to fetch this data is here:

def fetch_fundamental_data(api_key, stock_ticker, filing_date_gte, license_type="free"):
    """
    Fetch financial statement data for a given stock ticker using Polygon API.

    Parameters:
        api_key (str): Your Polygon API key.
        stock_ticker (str): Stock ticker symbol.
        filing_date_gte (str): Start date for the filing date filter.
        license_type (str): Type of license ("free" or "paid")

    Returns:
        list: A list containing the financial statement data.
    """
    # Create a REST client and authenticate with the API key
    client = RESTClient(api_key)

    # Initialize an empty list to hold the data
    data = []

    request_count = 0
    while True:
        try:
            # Fetch the financial statement data and add it to the list
            for t in client.vx.list_stock_financials(ticker=stock_ticker, filing_date_gte=filing_date_gte, limit=100):
                logging.info(f'{stock_ticker}: Got data from API {t}')
                data.append(t)

                # Check if the fetched data is empty
                if not data:
                    logging.warning(f"No data found for stock ticker {stock_ticker}. It may be an invalid symbol.")
                    return None

                request_count += 1

                # Log the latest filing date fetched
                latest_filing_date = t.filing_date  # Replace this with the actual attribute name for the date
                logging.info(f"Latest filing date fetched: {latest_filing_date}")

                if license_type == "free" and request_count >= 5:  # Check if the rate limit is reached
                    time.sleep(60)  # Pause for 60 seconds
                    request_count = 0  # Reset the request count

            break  # Exit the while loop if successful
        except PolygonAPIError as e:
            if "maximum requests per minute" in str(e):
                time.sleep(60)  # Pause for 60 seconds
            else:
                raise  # Re-raise the exception if it's not a rate-limit error

    return data

     cik       company_name    end_date filing_date fiscal_period  \

0 0000920760 LENNAR CORP /NEW/ 2023-08-31 2023-09-29 Q3
1 0000920760 LENNAR CORP /NEW/ 2023-05-31 2023-06-30 Q2
2 0000920760 LENNAR CORP /NEW/ 2023-02-28 2023-04-04 Q1
3 0000920760 LENNAR CORP /NEW/ 2022-11-30 2023-01-26 FY
4 0000920760 LENNAR CORP /NEW/ 2022-08-31 2022-10-04 Q3
5 0000920760 LENNAR CORP /NEW/ 2022-05-31 2022-07-01 Q2
6 0000920760 LENNAR CORP /NEW/ 2022-02-28 2022-04-01 Q1
7 0000920760 LENNAR CORP /NEW/ 2021-11-30 2022-01-28 FY
8 0000920760 LENNAR CORP /NEW/ 2021-08-31 2021-10-01 Q3
9 0000920760 LENNAR CORP /NEW/ 2021-05-31 2021-07-02 Q2
10 0000920760 LENNAR CORP /NEW/ 2021-02-28 2021-04-01 Q1
11 0000920760 LENNAR CORP /NEW/ 2020-11-30 2021-01-22 FY
12 0000920760 LENNAR CORP /NEW/ 2020-08-31 2020-10-01 Q3
13 0000920760 LENNAR CORP /NEW/ 2020-05-31 2020-07-06 Q2
14 0000920760 LENNAR CORP /NEW/ 2020-02-29 2020-04-07 Q1

fiscal_year current_liabilities equity_attributable_to_parent
0 2023 1.164958e+10 2.565662e+10
1 2023 -2.516112e+10 2.501514e+10
2 2023 -2.455529e+10 2.441826e+10
3 2022 1.374393e+10 2.410050e+10
4 2022 1.221234e+10 2.297728e+10
5 2022 -2.178977e+10 2.159826e+10
6 2022 -2.084743e+10 2.067906e+10
7 2021 1.221150e+10 2.081642e+10
8 2021 1.196465e+10 2.065019e+10
9 2021 -1.970210e+10 1.957611e+10
10 2021 -1.901745e+10 1.889625e+10
11 2020 1.183578e+10 1.799486e+10
12 2020 1.203494e+10 1.717210e+10
13 2020 -1.663262e+10 1.654270e+10
14 2020 -1.619338e+10 1.604460e+10

noncurrent_assets  noncurrent_liabilities  ...  \

0 0 0 ...
1 0 0 ...
2 0 0 ...
3 0 0 ...
4 0 0 ...
5 0 0 ...
6 0 0 ...
7 0 0 ...
8 0 0 ...
9 0 0 ...
10 0 0 ...
11 0 0 ...
12 0 0 ...
13 0 0 ...
14 0 0 ...

net_cash_flow_from_financing_activities  comprehensive_income_loss  \

0 -1.109284e+09 1.117160e+09
1 -5.745670e+08 8.783150e+08
2 -1.483463e+09 6.001590e+08
3 -1.277279e+09 4.652250e+09
4 -4.342220e+08 1.473036e+09
5 -1.322300e+08 1.322620e+09
6 -1.257886e+09 5.123420e+08
7 -2.404735e+09 4.456013e+09
8 -4.840860e+08 1.409349e+09
9 -1.881450e+08 8.370880e+08
10 -6.296880e+08 1.015967e+09
11 -2.446575e+09 2.466250e+09
12 -9.174200e+08 6.664180e+08
13 -1.587660e+08 5.174060e+08
14 -7.898040e+08 3.984520e+08

comprehensive_income_loss_attributable_to_parent  \

0 1.109204e+09
1 8.722670e+08
2 5.973850e+08
3 4.617874e+09
4 1.467686e+09
5 1.320818e+09
6 5.066080e+08
7 4.429575e+09
8 1.407019e+09
9 8.316790e+08
10 1.000427e+09
11 2.463733e+09
12 6.665930e+08
13 5.166160e+08
14 3.984060e+08

other_comprehensive_income_loss  basic_earnings_per_share  \

0 208000.0 3.87
1 573000.0 3.01
2 851000.0 2.06
3 3749000.0 15.74
4 342000.0 5.04
5 62000.0 4.50
6 3027000.0 1.70
7 -536000.0 14.28
8 131000.0 4.52
9 316000.0 2.66
10 -942000.0 3.20
11 -1303000.0 7.88
12 175000.0 2.13
13 -790000.0 1.66
14 -46000.0 1.27

operating_expenses      revenues  cost_of_revenue  gross_profit  symbol  

0 7.258891e+09 8.729603e+09 NaN NaN LEN
1 6.852312e+09 8.045151e+09 NaN NaN LEN
2 5.674155e+09 6.490429e+09 NaN NaN LEN
3 NaN NaN NaN NaN LEN
4 NaN NaN NaN NaN LEN
5 NaN NaN NaN NaN LEN
6 5.400582e+09 6.203516e+09 NaN NaN LEN
7 2.205224e+10 2.713068e+10 NaN NaN LEN
8 5.016908e+09 6.941403e+09 NaN NaN LEN
9 5.228150e+09 6.430245e+09 NaN NaN LEN
10 3.875609e+09 5.325468e+09 NaN NaN LEN
11 1.900665e+10 2.248885e+10 1.774076e+10 4.748090e+09 LEN
12 4.035370e+08 5.870254e+09 4.607704e+09 1.262550e+09 LEN
13 3.857330e+08 5.287373e+09 4.225063e+09 1.062310e+09 LEN
14 4.254360e+08 4.505337e+09 3.656349e+09 8.489880e+08 LEN

[15 rows x 29 columns]

[justinpolygon]

Thanks for the heads up @JeremyWhittaker. Thank for the very detailed write up as it helps us track things down quickly. After taking a look, this is more of a data issue/gap than a client library issue so I pinged the backend data team and they will check it out. I'll keep you posted.

[JeremyWhittaker]

Thanks for the heads up @JeremyWhittaker. Thank for the very detailed write up as it helps us track things down quickly. After taking a look, this is more of a data issue/gap than a client library issue so I pinged the backend data team and they will check it out. I'll keep you posted.

Appreciate it. I just signed up for your service and I'm trying to find reliable data to analyze, When I run across stuff like this it makes me start to question all of my output.

[JeremyWhittaker]

Same symbol, huge chunks of data missing from this metric as well:

INFO:root: fiscal_period fiscal_year end_date filing_date cost_of_revenue
0 Q3 2023 2023-08-31 2023-09-29 NaN
1 Q2 2023 2023-05-31 2023-06-30 NaN
2 Q1 2023 2023-02-28 2023-04-04 NaN
3 FY 2022 2022-11-30 2023-01-26 NaN
4 Q3 2022 2022-08-31 2022-10-04 NaN
5 Q2 2022 2022-05-31 2022-07-01 NaN
6 Q1 2022 2022-02-28 2022-04-01 NaN
7 FY 2021 2021-11-30 2022-01-28 NaN
8 Q3 2021 2021-08-31 2021-10-01 NaN
9 Q2 2021 2021-05-31 2021-07-02 NaN
10 Q1 2021 2021-02-28 2021-04-01 NaN
11 FY 2020 2020-11-30 2021-01-22 1.774076e+10
12 Q3 2020 2020-08-31 2020-10-01 4.607704e+09
13 Q2 2020 2020-05-31 2020-07-06 4.225063e+09
14 Q1 2020 2020-02-29 2020-04-07 3.656349e+09
15 FY 2019 2019-11-30 2020-01-27 1.802340e+10
16 Q3 2019 2019-08-31 2019-10-08 4.771425e+09
17 Q2 2019 2019-05-31 2019-07-03 4.524303e+09
18 Q1 2019 2019-02-28 2019-04-08 3.142003e+09
19 FY 2018 2018-11-30 2019-01-28 1.688282e+10
20 Q3 2018 2018-08-31 2018-10-09 4.614666e+09
21 Q2 2018 2018-05-31 2018-07-06 4.619019e+09
22 Q1 2018 2018-02-28 2018-04-09 2.464163e+09
23 FY 2017 2017-11-30 2018-01-25 1.021241e+10
24 Q3 2017 2017-08-31 2017-10-10 2.611065e+09
25 Q2 2017 2017-05-31 2017-06-30 2.645017e+09
26 Q1 2017 2017-02-28 2017-04-10 1.918263e+09
27 FY 2016 2016-11-30 2017-01-20 8.754335e+09
28 Q3 2016 2016-08-31 2016-10-04 2.282218e+09
29 Q2 2016 2016-05-31 2016-07-01 2.184292e+09
30 Q1 2016 2016-02-29 2016-04-06 1.594718e+09
31 FY 2015 2015-11-30 2016-01-22 NaN
32 Q3 2015 2015-08-31 2015-10-09 NaN
33 Q2 2015 2015-05-31 2015-07-02 NaN
34 Q1 2015 2015-02-28 2015-04-03 NaN
35 FY 2014 2014-11-30 2015-01-23 NaN
36 Q3 2014 2014-08-31 2014-10-03 NaN
37 Q2 2014 2014-05-31 2014-07-03 NaN
38 Q1 2014 2014-02-28 2014-04-09 NaN
39 FY 2013 2013-11-30 2014-01-28 NaN
40 Q3 2013 2013-08-31 2013-10-10 NaN
41 Q2 2013 2013-05-31 2013-07-10 NaN
42 Q1 2013 2013-02-28 2013-04-09 NaN
43 FY 2012 2012-11-30 2013-01-29 NaN
44 Q3 2012 2012-08-31 2012-10-10 NaN
45 Q2 2012 2012-05-31 2012-07-10 NaN
46 Q1 2012 2012-02-29 2012-04-09 NaN
47 FY 2011 2011-11-30 2012-01-30 NaN
48 Q3 2011 2011-08-31 2011-10-11 NaN
49 Q2 2011 2011-05-31 2011-07-11 NaN
50 Q1 2011 2011-02-28 2011-04-11 NaN
51 Q3 2010 2010-08-31 2010-10-08 NaN