Download Instagram with yt-dlp

- Bugs fixed

Signed-off-by: anasty17 <e.anastayyar@gmail.com>

.gitignore

@@ -6,6 +6,8 @@ data*
 .idea
 *.json
 *.pickle
+.netrc
+log.txt
 authorized_chats.txt
 sudo_users.txt
 accounts/*

.netrc

@@ -1 +0,0 @@
-

README.md

@@ -112,12 +112,13 @@ Fill up rest of the fields. Meaning of each field is discussed below:
 <details>
     <summary><b>Click Here For More Details</b></summary>
 
-- `ACCOUNTS_ZIP_URL`: Only if you want to load your Service Account externally from an Index Link. Archive the accounts folder to a zip file. Fill this with the direct link of that file.
+- `ACCOUNTS_ZIP_URL`: Only if you want to load your Service Account externally from an Index Link or by any direct download link NOT webpage link. Archive the accounts folder to a zip file. Fill this with the direct download link of that file. If index need authentication so add direct download as shown below:
+  - `https://username:passwdord@example.workers.dev/...`
 - `TOKEN_PICKLE_URL`: Only if you want to load your **token.pickle** externally from an Index Link. Fill this with the direct link of that file.
 - `MULTI_SEARCH_URL`: Check `drive_folder` setup [here](https://github.com/anasty17/mirror-leech-telegram-bot/tree/master#multi-search-ids). Write **drive_folder** file [here](https://gist.github.com/). Open the raw file of that gist, it's URL will be your required variable. Should be in this form after removing commit id: https://gist.githubusercontent.com/username/gist-id/raw/drive_folder
 - `YT_COOKIES_URL`: Youtube authentication cookies. Check setup [Here](https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl). Use gist raw link and remove commit id from the link, so you can edit it from gists only.
-- `NETRC_URL`: Use this incase you want to deploy heroku branch without filling `UPSTREAM_REPO` variable, since after restart this file will cloned from github as empty file. Use gist raw link and remove commit id from the link, so you can edit it from gists only.
-- `DATABASE_URL`: Your Database URL. See [Generate Database](https://github.com/anasty17/mirror-leech-telegram-bot/tree/master#generate-database) to generate database (**NOTE**: If you use database you can save your Sudo ID permanently using `/addsudo` command).
+- `NETRC_URL`: To create .netrc file contains authentication for aria2c and yt-dlp. Use gist raw link and remove commit id from the link, so you can edit it from gists only. **NOTE**: After editing .nterc you need to restart the docker or if deployed on heroku so restart dyno in case your edits related to aria2c authentication.
+- `DATABASE_URL`: Your Database URL. See [Generate Database](https://github.com/anasty17/mirror-leech-telegram-bot/tree/master#generate-database) to generate database (**NOTE**: If you use database you can save your Sudo ID permanently).
 - `AUTHORIZED_CHATS`: Fill user_id and chat_id (not username) of groups/users you want to authorize. Separate them with space, Examples: `-0123456789 -1122334455 6915401739`.
 - `SUDO_USERS`: Fill user_id (not username) of users whom you want to give sudo permission. Separate them with space, Examples: `0123456789 1122334455 6915401739` (**NOTE**: If you want to save Sudo ID permanently without database, you must fill your Sudo Id here).
 - `IS_TEAM_DRIVE`: Set to `True` if `GDRIVE_FOLDER_ID` is from a Team Drive else `False` or Leave it empty. `Bool`
@@ -401,18 +402,21 @@ MyTdName2 folderID/tdID IndexLink(if available)
 ```
 -----
 
-## Yt-dlp and Index Authentication Using .netrc File
-For using your premium accounts in yt-dlp or for protected Index Links, edit the netrc file according to following format:
+## Yt-dlp and Aria2c Authentication Using .netrc File
+For using your premium accounts in yt-dlp or for protected Index Links, create .netrc file according to following format:
+**Note**: Create .netrc and not netrc, this file will be hidden, so view hidden files to edit it after creation.
 ```
 machine host login username password my_password
 ```
+Example: machine instagram login anas.tayyar password mypassword
+**Instagram Note**: You must login even if you want to download public posts and after first try you must confirm that this was you logged in from different ip(you can confirm from phone app).
 **Note**: For `youtube` authentication use [cookies.txt](https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl) file.
 
 For Index Link with only password without username, even http auth will not work, so this is the solution.
 ```
 machine example.workers.dev password index_password
 ```
-Where host is the name of extractor (eg. Twitch). Multiple accounts of different hosts can be added each separated by a new line.
+Where host is the name of extractor (eg. instagram, Twitch). Multiple accounts of different hosts can be added each separated by a new line.
 
 -----
 

alive.py

@@ -15,8 +15,8 @@ PORT = os.environ.get('PORT', None)
 if PORT is not None and BASE_URL is not None:
     while True:
         try:
+            requests.get(BASE_URL).status_code
             time.sleep(600)
-            status = requests.get(BASE_URL).status_code
         except RequestException as e:
             logging.error(str(e))
             continue

bot/__init__.py

@@ -62,10 +62,13 @@ PORT = os.environ.get('PORT', SERVER_PORT)
 web = subprocess.Popen([f"gunicorn wserver:start_server --bind 0.0.0.0:{PORT} --worker-class aiohttp.GunicornWebWorker"], shell=True)
 alive = subprocess.Popen(["python3", "alive.py"])
 nox = subprocess.Popen(["qbittorrent-nox", "--profile=."])
+if not os.path.exists('.netrc'):
+    subprocess.run(["touch", ".netrc"])
 subprocess.run(["chmod", "600", ".netrc"])
 subprocess.run(["chmod", "+x", "aria.sh"])
 subprocess.run(["./aria.sh"], shell=True)
 time.sleep(0.5)
+
 Interval = []
 DRIVES_NAMES = []
 DRIVES_IDS = []

bot/helper/mirror_utils/download_utils/aria2_download.py

@@ -15,7 +15,7 @@ class AriaDownloadHelper:
     @new_thread
     def __onDownloadStarted(self, api, gid):
         if STOP_DUPLICATE or TORRENT_DIRECT_LIMIT is not None or ZIP_UNZIP_LIMIT is not None:
-            sleep(1)
+            sleep(2)
             dl = getDownloadByGid(gid)
             download = api.get_download(gid)
             if STOP_DUPLICATE and dl is not None and not dl.getListener().isLeech:

bot/helper/mirror_utils/download_utils/qbit_downloader.py

@@ -1,7 +1,3 @@
-# Implement By - @anasty17 (https://github.com/SlamDevs/slam-mirrorbot/commit/0bfba523f095ab1dccad431d72561e0e002e7a59)
-# (c) https://github.com/SlamDevs/slam-mirrorbot
-# All rights reserved
-
 import os
 import random
 import string
@@ -249,10 +245,7 @@ def get_confirm(update, context):
 
 def get_hash_magnet(mgt):
     if mgt.startswith('magnet:'):
-        try:
-            mHash = re.search(r'xt=urn:btih:(.*)&dn=', mgt).group(1)
-        except:
-            mHash = re.search(r'xt=urn:btih:(.*)', mgt).group(1)
+        mHash = re.search(r'(?<=xt=urn:btih:)[a-zA-Z0-9]+(?=&dn=|.?)', mgt).group(0)
         return mHash.lower()
 
 def get_hash_file(path):

bot/helper/mirror_utils/download_utils/youtube_dl_download_helper.py

@@ -56,10 +56,9 @@ class YoutubeDLHelper(DownloadHelper):
         self.opts = {'progress_hooks': [self.__onDownloadProgress],
                      'logger': MyLogger(self),
                      'usenetrc': True,
-                     'continuedl': True,
                      'embedsubtitles': True,
                      'prefer_ffmpeg': True,
-                     'cookiefile': 'cookies.txt' }
+                     'cookiefile': 'cookies.txt'}
 
     @property
     def download_speed(self):

bot/helper/mirror_utils/upload_utils/pyrogramEngine.py

@@ -11,9 +11,9 @@ from bot.helper.ext_utils.fs_utils import take_ss, get_media_info
 LOGGER = logging.getLogger(__name__)
 logging.getLogger("pyrogram").setLevel(logging.ERROR)
 
-VIDEO_SUFFIXES = ("MKV", "MP4", "MOV", "WMV", "3GP", "MPG", "WEBM", "AVI", "FLV", "M4V")
+VIDEO_SUFFIXES = ("MKV", "MP4", "MOV", "WMV", "3GP", "MPG", "WEBM", "AVI", "FLV", "M4V", "GIF")
 AUDIO_SUFFIXES = ("MP3", "M4A", "M4B", "FLAC", "WAV", "AIF", "OGG", "AAC", "DTS", "MID", "AMR", "MKA")
-IMAGE_SUFFIXES = ("JPG", "JPX", "PNG", "GIF", "WEBP", "CR2", "TIF", "BMP", "JXR", "PSD", "ICO", "HEIC", "JPEG")
+IMAGE_SUFFIXES = ("JPG", "JPX", "PNG", "WEBP", "CR2", "TIF", "BMP", "JXR", "PSD", "ICO", "HEIC", "JPEG")
 
 
 class TgUploader:

bot/modules/shell.py

@@ -4,26 +4,24 @@ from telegram import ParseMode
 from telegram.ext import CommandHandler
 from bot.helper.telegram_helper.filters import CustomFilters
 from bot.helper.telegram_helper.bot_commands import BotCommands
+from bot.helper.telegram_helper.message_utils import sendMessage
 
 
 def shell(update, context):
     message = update.effective_message
     cmd = message.text.split(' ', 1)
     if len(cmd) == 1:
-        message.reply_text('No command to execute was given.')
-        return
+        return sendMessage('No command to execute was given.', context.bot, update)
     cmd = cmd[1]
-    process = subprocess.Popen(
-        cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
-    stdout, stderr = process.communicate()
+    process = subprocess.run(cmd, capture_output=True, shell=True)
     reply = ''
-    stderr = stderr.decode()
-    stdout = stdout.decode()
-    if stdout:
-        reply += f"*Stdout*\n`{stdout}`\n"
+    stderr = process.stderr.decode('utf-8')
+    stdout = process.stdout.decode('utf-8')
+    if len(stdout) != 0:
+        reply += f"*Stdout*\n<code>{stdout}</code>\n"
         LOGGER.info(f"Shell - {cmd} - {stdout}")
-    if stderr:
-        reply += f"*Stderr*\n`{stderr}`\n"
+    if len(stderr) != 0:
+        reply += f"*Stderr*\n<code>{stderr}</code>\n"
         LOGGER.error(f"Shell - {cmd} - {stderr}")
     if len(reply) > 3000:
         with open('shell_output.txt', 'w') as file:
@@ -34,10 +32,12 @@ def shell(update, context):
                 filename=doc.name,
                 reply_to_message_id=message.message_id,
                 chat_id=message.chat_id)
+    elif len(reply) != 0:
+        sendMessage(reply, context.bot, update)
     else:
-        message.reply_text(reply, parse_mode=ParseMode.MARKDOWN)
+        sendMessage('No Reply', context.bot, update)
 
 
-SHELL_HANDLER = CommandHandler(BotCommands.ShellCommand, shell, 
+SHELL_HANDLER = CommandHandler(BotCommands.ShellCommand, shell,
                                                   filters=CustomFilters.owner_filter, run_async=True)
 dispatcher.add_handler(SHELL_HANDLER)