Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Help - no source for stock prices!

2

Comments

  • Morning Vidas

    It wasn't so important that I needed to do the update late last night, so I waited until now (13:20 our time, 5 hours after the markets opened) to re-run the scripts.

    As with yesterday, I am still getting the error message, though it seems to be for fewer stocks (it doesn't seem to have an issue now with, e.g, BOY or BSIF) - please see the attached image vs yesterday's attachment. I'm not sure why some current holdings are listed as having duplicate records and others aren't?

    To your comments about what happened when I ran the scrip:

    1) The intraday folder is empty other than two indices I have (INDEXFTSE:SMXX and INDEXFTSE:AXX) in a \QuotesIntraDay\googleintraday.txt file. It has not extracted any yahoo intraday quotes for my XXX.L stock holdings.

    2) The quotes folder does contain, eg, _ATYM.L_.tx file, but it has no content at all (presumably because of the min date being 2017-10-05, as markets are not yet shut for today)

    3) As you said previously, the script seems to have generated the archive files in the quotes folder for the data I downloaded from my other source, so that bits seems to be working ok.

    As yet, I have not ventured to refresh the data in the PortfolioSlicer workbook as I thought best to make sure the scripts are working properly and without issue, rather than pulling the wrong / erroneous data into excel PowerPivot data model.

    Look forward to hearing your thoughts.
  • Could you please zip and send me content of your PSData and Quotes,QuotesIntraday folders and psConfig.txt. I want to investigate this. My email is posted on this page: http://www.portfolioslicer.com/contact.html
  • Thanks for sending me your files. There are few things happening at the same time that needs to be fixed:
    1. Download the latest script - you are still using scripts with the bug. Specifically, a bug that affects you is in the script psCheckFiles.ps1 and it should have a date of 2017-09-18. After updating script you will see that YahooIntraday file was created and has data that cause an issue.
    2. The naming of the files in your \Quotes problem is not right and that cause duplicate issues. Let's take at one symbol - SDV.L. Quotes you created MUST be in the file _SDV.L_.txt and not in _SDV_.txt and inside symbol must be SDV.L and not SDV. That way should be even if you later rename symbol SDV.L to SDV.
    To explain this issue: when script checks until when you have quotes, it will look into file _SDV.L_.txt. In your case that file is not found, so script assumes no quotes exits at all. Then intraday script gets data for latest trade, that happened to be 2017-10-03 and that create duplicates.

    So, please address these issues and let me know if you have any more issues.

  • Hi Vidas

    Good news - the scripts ran ok without error messages and the quote files seem fine!!

    However, I have noticed that my values were all over the place. I'm pretty certain this is because of the need for FactorHistory, FactorIntraDay (as yahoo generally uses pence for its share price data, not pounds). So I have added 0.01,0.01 in the relevant place in my yahoo list of current symbols. However, when discussing point 7 above we said that old (historic) holdings could be removed from the psConfig.txt file - but if I do this, I do not have a way to tell portfolio slicer that it needs to convert the quote file data for historic holdings into pounds, do I?

    In other words, because all of my stocks (current and historic) are in pence, do I need to keep all of them in the psConfig.txt file with the FactorHistory,FactorIntraDay fields completed 0.01,0.01.

    If I need to put my historic stocks back in the file, I suppose I can always move to the bottom of the list and add a relevant MaxData field to stop yahoo searching for the most recent price data. But is there a better way of doing the same thing without keeping them all there?

    Thanks

    Laurence
  • Laurence,

    If for your old holding you need to apply Factor, you need to keep symbol info in psConfig.txt, just add MinDate and MaxDate so quotes will not be requested, but Factor fill be applied when Quotes.csv file will be created. Where did you get old holdings from? Yahoo?

    Another option - you can edit your existing old holding quotes and apply factors and save back. You can do that in Excel, if you know Excel well.
  • Hi Vidas

    Sorry for the delay responding - I've been away without regular internet access.

    My old holdings were obtained from this resource: https://www.sharescope.co.uk/sharescope.jsp. I use it for my stock screening and I discovered that it has great export functionality.

    However, I can't export in exactly the format I need - it will do all shares in one csv file, or one csv file for each holding. The latter case would be perfect except that there are two columns only (date and close price), but no EPIC (the software licensor argues that this info is in the file name, so not needed within the file itself). So I had to manually edit 100+ csv files to add the EPIC column! Easy to do, esp. with excel shortcuts keys, but very boring and time-consuming for so many text files.

    1. Is there an application that can help people like me (who cannot write scrip code at all) create scripts to do a simple task such as the above automatically?? This would be my best solution.

    In the absence of such a script solution, I have configured my psconfig.txt file so that going forward yahoo price data is downloaded for my current holdings. This seems to be working ok, but I noticed a few things that are causing problems with my mutual fund holdings (UK closed-ended investment companies):

    2. EOD pricing seems to be in GBP, whereas the intraday is GBX! Can't understand why yahoo does the switch, but I think the psconfig.txt file provides for intraday and historic price factoring, so I can overcome this.

    3. more troublesome however is an inconsistency in the yahoo historic data download compared to the historic prices shown on the yahoo website. For instance, compare https://finance.yahoo.com/quote/JMF.L/history?p=JMF.L with the end of my quote file _JMF.L_.txt:

    2017-10-10,12.322700,JMF.L
    2017-10-11,11.200000,JMF.L
    2017-10-12,11.260000,JMF.L

    2017-10-11 is totally different, and I cannot understand why.

    Further, if I look at the share price chart, I can clearly see that the share has not yet reached £12 and so both sources are clearly incorrect in some respects. I use this example, but I discovered similar irregularities for about 10 such mutual fund holdings.

    I should clarify that the intraday pricing looks fine - it is just the historic pricing for these mutual funds that is completely / partially incorrect.

    Per my question 1 above, a simple script file to download the historic data for these few holdings would solve this yahoo glitch I am facing. Or is there something else happening here that I'm not aware of?

    4. Finally, unrelated to the quotes, I am struggling to understand how the profit % figures are calculated. If I take my biggest winner of 2017 as an example, my cost basis was 5990, current value is 11,942 and I have received dividends of 263. My profit is therefore 6215 (11942-5990+263), giving me a profit % of 103.8%; however, the PS workbook states it is 145.5%?

    Any ideas on the above please?

    Thanks

    Laurence
  • 1. I am not sure if such application exists. I would do everything in Excel and then save quotes in the files, but it would have to be separate files for each quote. but - separate file for each symbol (tha tis not actively used or requires Factor) is a requirement for archive script to work properly, so if you turn archiving off, you could keep all quotes in one file.
    2. Yes, your right - tHat is why psConfig.txt for each symbol allow to specify separate intraday and history factor.
    3. Try different quote source - try stooq and see if data is any better. Read more new documentation: http://portfolioslicer.com/scripts/quotes-datasource.html
    4. Profit % is calculated monthly and then results are "linked". So to test if Profit % is right, you should create a pivot table with monthly data (Cap gain, Profit, Profit %) for one symbol. Some info about "linking" is here: https://en.wikipedia.org/wiki/True_Time-Weighted_Rate_of_Return. So build monthly report and check results, they might explai why you see not what is expected.
  • Thanks Vidas, I will investigate
  • Hi Vidas

    Re 3 above, I have explored the different quote sources extensively. Stooq seems to be very good actually, while Alpha Vantage appears completely wrong for my investment company (similar to mutual fund) holdings.

    However, I have historic info for these holdings from my alternative source (now converted to format XXX.L), while Stooq is in format XXX.UK. I think this would be fine if I used the symbol rename function in psconfig.txt and could start Stooq quote extractions from the date immediately following the last one in my XXX.L quote files - however I cannot see the configuration option in psconfig.txt to allow me to start Stooq quote retrieval from a certain mindate. And there is no comment about it here: http://www.portfolioslicer.com/scripts/quotes-datasource.html.

    I assume therefore I should amend my old XXX.L quote files to XXX.UK in excel so that Stooq can append new stock price information to the existing data I have collected from my alternative stock price source.

    Is that correct?

    Thanks

    Laurence
  • edited October 2017
    In the link you specified http://www.portfolioslicer.com/scripts/quotes-datasource.html at the very bottom I listed options:

    How to change the quote source

    Sometimes you will need to replace your symbol quote source. When you do that, you must make sure that final created PSData\Quotes.csv file has no duplicate values and used symbol is consistent.

    If the symbol is exactly the same between 2 sources (example AAPL in Yahoo Finance and Stooq), then you can simply edit psConfig.txt and move symbol from one section to another (example from Yahoo section to AlphaVantage section). But if the Symbol changed between sources (example AAPL in Yahoo Finance and AAPL.US in Stooq), then there is a little bit more work involved in changing data source.

    Let’s say you have symbol “AAPL” and you get quotes from Yahoo Finance website. From Oct 1st, 2017 you want to change quote source for this symbol to Stooq. You already have symbol AAPL defined in Portfolio Slicer Excel workbook as “AAPL”. Here is how you can accomplish this:

    Option 1 - re-extract all data. In psConfig.txt file move AAPL from Yahoo section to Stooq section and change AAPL symbol to AAPL.US. In the folder “\Quotes\” delete (backup first!!!) files _AAPL_.txt and _AAPL__Archive.txt. To SymbolRename section add string “AAPL.US,AAPL”. Re-run extract for external data.
    Option 2 - rename existing data. In psConfig.txt file move AAPL from Yahoo section to Stooq section and change AAPL symbol to AAPL.US. In the folder “\Quotes\” rename files _AAPL_.txt to _AAPL.US_.txt and file _AAPL__Archive.txt to _AAPL.US__Archive.txt. Open these files in the notepad and replace string “AAPL” with “AAPL.US”. To SymbolRename section add string “AAPL.US,AAPL”. Re-run extract for external data. In such setup scripts for Stooq will just append data to existing quote files.
    Option 3 - combine old and new data. In psConfig.txt file move AAPL from Yahoo section to Stooq section and change AAPL symbol to AAPL.US and add SymbolMinDate, like this: “AAPL.US,2017-10-01”. In the folder “\Quotes\” copy files _AAPL_.txt and _AAPL__Archive.txt to the subfolder “\Quotes\Manual\“, open each file and make sure there are no quotes after 2017-09-30. To SymbolRename section add string “AAPL.US,AAPL”. Re-run extract for external data. In such setup, old quotes are moved to another folder and new quotes are extracted just from specified date and then these quotes are merged when they are renamed.


    Let me know if you still have issue, I can provide more details, but let me know what option you want to use 1,2 or 3.
  • Hi Vidas

    Sorry, I missed that.

    1. Option 3 looks best for me and I believe I have already achieved this for one mutual fund holding I had, albeit I didn't have a manual folder. I will pursue this route for now.

    2. If I have understood it correctly, Stooq doesn't offer intraday pricing - so is there anyway to retrieve yahoo intraday only (i.e. not historic data) for these holdings? (I know this sounds peculiar, but the yahoo historic data seems accurate for intraday but very inaccurate for historic for these holdings, so I must use Stooq as an alternative source for the historic data.)

    3. Finally, last year I held an investment company (ECWO) which was reconstituted. The underlying liquid holdings of the investment company were reclassified as EGL shares (on a 1:1 basis) and the illiquid holdings were reclassified as EFR shares (on a 4 ECWO : 1 EFR basis). Further, I elected to receive 97.6% of what would have been my EGL entitlement in cash (based on NAV at the date), as was offered by the corporate action.

    Can you advise how I should record the above in my transaction sheet?! I think the cash payout could simply be done as a notional sale of EGL shares, but how do I account for a split of ECWO into two new stocks (EGL and EFR)?

    Thanks again

    Laurence
  • Hi Laurence,

    1. OK, let me know if you will have any issues with this option.
    2. You could trick system to extract Yahoo intraday quotes. For that you have to make sure that in .bat file you remove call to GetQuotes-Yahoo.ps1 script (add REM and space at the start of the line) but leave GetQuotes-YahooIntraday.ps1 script. Add symbol to both Yahoo and Stooq sections in psConfig.txt, if Yahoo symbol is different, add line in rename symbol section.
    3. You can achieve this buy:
    a) capture what is Cost Base for your symbol ECWO.
    b) figure out how much of value of ECWO will go to EGL and ERF, example 60% and 40%
    c) add transaction SymbolTransferOut for ECWO symbol with price at the point of split
    d) add transaction SymbolTransferIn for EGL, make sure that total value is equal 60% of ECWO, so you will have to calculate price per share. Calculate cost base as 60% of ECWO cost base and specify that in CostBasisOverride field for this transaction.
    e) add transaction SymbolTransferIn for ERF, make sure that total value is equal 40% of ECWO. Calculate cost base as 40% of ECWO cost base and specify that in CostBasisOverride field for this transaction.
    f) cash payment 97.6% can be handled either as dividend, or you can choose to first transfer in more shares and record sale for that amount. Or you can do this as sale of original ECWO one day before SymbolTansferOut - that would probably make all calculations more natural. You have many options..
  • Hi Vidas

    Getting very close now, I think.

    Re. 1 above, I have performed the move and it has worked without errors when UpdatePSData.bat was been run. However, my quote files in the new manual folder (i.e. for the quotes up to 2017-09-30, per your example) need a factor conversion (they are in pence, not pounds).

    Since these quote files are static and do not appear in psconfig.txt file, I cannot apply the factor adjustment there.

    Is there any programme that can do a batch adjustment of the price data in these files for me - i.e. do an effective division of all the historic price data by 100. (I have considered excel, but, as there are a lot of these files, to convert them all to csv files and then apply to a formula in each file to divide the price column by 100, and then convert back to .txt, will take a long time.)

    Note that certain historic data has a decimal place, while others don't - eg.:

    2013-07-11,103,BSIF.L
    2013-07-31,101,BSIF.L
    2013-08-30,102.375,BSIF.L
    2013-09-30,103.5,BSIF.L

    I was hoping to find a programme which could find the first '.' and move it two places to the left, but because the '.' doesn't appear universally that won't work here.

    Any ideas??

    2 and 3 above to be attempted once I have resolved the above.

    Thanks as ever

    Laurence
  • There is no such program, if you need to divide by 100, then your best choice would be re-format data in Excel. For file where you have . as decimal separator, you could use Notepad++ and record macro to move . two positions left, but it might not work perfectly. I would do all that data manipulation in Excel, then save in csv.
  • Hi Vidas

    I managed to find someone who has helped me by creating a small executable file to convert the c.100 historic quote files I had from my alternative source from GBX to GBP. It's a great result as this has allowed me to clean up my Quotes folder (adopting the manual sub-foldering you suggested) and PSConfig.txt file as I do not need to list these historic holdings in my yahoo list simply for factoring purposes.

    A couple of last questions from me -

    1. I note the demise of yahoo intraday and comments on this from you/others on other threads. But do you know if this is likely to be temporary or permanent? If the latter, I'm thinking about whether I should use google for my intraday, and Stooq for my historic data going forward...

    2. Re. back-ups, am I right in thinking that I need to do this only for my main portfolio slicer directory (including sub-folder content), and that I do not need to do it for my external files (those saved in c:\PortfolioSlicer\PSData) as these are simply the bi-product of my source data which is in the main portfolio slicer directory?

    Thanks

    Laurence

Sign In or Register to comment.