# Databricks notebook source
# MAGIC %md
# MAGIC # CSR reports
# MAGIC
# MAGIC Any large scale organisation is now facing tremendous pressure from their shareholders to disclose more information about their environmental, social and governance strategies. Typically released on their websites on a yearly basis as a form of a PDF document, companies communicate on their key ESG initiatives across multiple themes such as how they value their employees, clients or customers, how they positively contribute back to society or even how they reduce (or commit to reduce) their carbon emissions. Consumed by third parties agencies, these reports are usually consolidated and benchmarked across industries to create ESG metrics. In this notebook, we would like to programmatically access 40+ ESG reports from top tier financial services institutions and learn key ESG initiatives across different topics.

# COMMAND ----------

# MAGIC %run ./config/esg_config

# COMMAND ----------

# MAGIC %md
# MAGIC ## Extract PDF
# MAGIC In this section, we search for publicly available corporate sustainability documents from publicly traded organizations ([example](https://home.barclays/content/dam/home-barclays/documents/citizenship/ESG/Barclays-PLC-ESG-Report-2019.pdf)). Instead of going through each company website, one could access information from [responsibilityreports.com](https://www.responsibilityreports.com), manually browsing for content or automatically scraping new records (please check [T&Cs](https://www.responsibilityreports.com/Disclaimer)). For the purpose of this exercise, we only provide URLs to a few financial services organizations listed below. Please be aware that the quality of analytics derived in that solution strongly depends on the amount of PDF documents to learn from. We recommend training models with 100+ PDFs to extract meaningful insights (even if the end goal is only to classify a few documents).

# COMMAND ----------

from io import StringIO
import pandas as pd

csv_str = """organisation,url
discover,https://www.responsibilityreports.com/Click/2357
equifax,https://www.responsibilityreports.com/Click/1346
canadian imperial bank,https://www.responsibilityreports.com/Click/1894
citigroup,https://www.responsibilityreports.com/Click/1515
eurobank,https://www.responsibilityreports.com/Click/3126
jpmorgan chase,https://www.responsibilityreports.com/Click/1278
keybank,https://www.responsibilityreports.com/Click/1599
laurentian bank of canada,https://www.responsibilityreports.com/Click/1918
national australia bank,https://www.responsibilityreports.com/Click/1555
pnc,https://www.responsibilityreports.com/Click/1829
standard chartered,https://www.responsibilityreports.com/Click/2781
tcf financial,https://www.responsibilityreports.com/Click/1669
wells fargo,https://www.responsibilityreports.com/Click/1904
ameriprise,https://www.responsibilityreports.com/Click/1784
lazard,https://www.responsibilityreports.com/Click/1429
capital one,https://www.responsibilityreports.com/Click/1640
goldman sachs,https://www.responsibilityreports.com/Click/1496"""

esg_pdf = pd.read_csv(StringIO(csv_str))
display(esg_pdf)

# COMMAND ----------

# MAGIC %md
# MAGIC We download text content for online CSR disclosures using the `PyPDF2` library. Please note that you will need to support outbound HTTP access from your databricks workspace. Although having a central place where to source data from (such as [responsibilityreports.com](https://www.responsibilityreports.com)) minimizes the amount of firewall rules to enable, this approach comes at a price: it prevents user from distributing that scraping logic across multiple executors. Just like many web scraping processes, please proceed with extra caution and refer to responsibilityreports.com [T&Cs](https://www.responsibilityreports.com/Disclaimer) before doing so.

# COMMAND ----------

# MAGIC %run ./utils/esg_utils

# COMMAND ----------

esg_pdf['content'] = esg_pdf['url'].apply(get_csr_content)
display(esg_pdf)

# COMMAND ----------

# MAGIC %md
# MAGIC PDFs are highly unstructured by nature with text that is often scattered across multiple lines, pages, columns. From a simple set of regular expressions to a more complex NLP model (we use a `NLTK` trained pipeline), we show how to extract clean sentences from raw text documents. Not defined as a silver bullet but more as a working example, feel free to extend this framework further.

# COMMAND ----------

esg_df = spark.createDataFrame(esg_pdf)
esg_df = esg_df.withColumn('statement', F.explode(extract_statements('content'))).drop('content').cache()
esg_df.write.mode('overwrite').format('delta').option('path', config['csr_raw_path']).saveAsTable(config['csr_raw_table'])

# COMMAND ----------

esg_df = spark.read.table(config['csr_raw_table'])
display(esg_df)

# COMMAND ----------

# MAGIC %md
# MAGIC ## Organization contribution
# MAGIC Some organizations may be mentioned in the news via their common names (e.g. IBM) rather than their legal entities (e.g. International Business Machines Corporation), resulting in an overall poor match later when joining our datasets "as-is". Furthermore, many organizations are related to others, such as portfolio companies belonging to a same holding and mentioned in both public disclosure and news. Analysing ESG initiatives must take the global dynamics of a business into account. For that purpose, we will use NLP techniques to extract mentions of any organizations in a given CSR report and link those "named entities" to better capture ESG related news we know are relevant for any given organization.

# COMMAND ----------

from pyspark.sql.functions import pandas_udf
from typing import Iterator
import pandas as pd

@pandas_udf('array<string>')
def extract_organizations_udf(content_series_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
  nlp = load_spacy()
  for content_series in content_series_iter:
    yield content_series.map(lambda text: extract_organizations(text, nlp))

# COMMAND ----------

org_df = esg_df.select(F.col('organisation'), F.explode(extract_organizations_udf(F.col('statement'))).alias('alias')).filter(F.length('alias') > 0)
org_df.write.format('delta').mode('overwrite').option('path', config['csr_org_path']).saveAsTable(config['csr_org_table'])
organisations = set(list(org_df.toPandas().alias.apply(clean_org_name)) + list(org_df.toPandas().organisation))

# COMMAND ----------

# MAGIC %md
# MAGIC ## Learn topics
# MAGIC
# MAGIC In this section, we apply [latent dirichlet allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) to learn topics descriptive to CSR reports. Please note that the goal is to set the right foundations with a baseline model that can be further extended with different NLP techniques. We want to be able to better understand and eventually sumarize complex CSR reports into a specific ESG related themes (such as 'valuing employees'). Before doing so, we may want to consider additional stopwords that you may want to change to accomodate different industries or domain specific language

# COMMAND ----------

import gensim
from gensim.parsing.preprocessing import STOPWORDS

# add company names as stop words
org_stop_words = []
for organisation in organisations:
    for t in organisation.split(' '):
        org_stop_words.append(t)

# our list contains all english stop words + companies names + specific keywords
stop_words = STOPWORDS.union(org_stop_words)

# COMMAND ----------

esg_df = esg_df.withColumn('lemma', lemmatize_text('statement')).filter(F.length('lemma') > 255)
corpus = esg_df.select('lemma').toPandas().lemma

# COMMAND ----------

# MAGIC %md
# MAGIC The challenge of topic modelling is to extract good quality of topics that are clear and meaningful. This depends heavily on the quality of text preprocessing (above), the amount of data to learn from and the strategy of finding the optimal number of topics. With more data (more PDFs), we may learn more meaningful insights. With industry specific ESG reports, we may learn industry specific ESG initiatives as opposition to broader catagories. We highly recommend starting small with the following code snippet and further extend this framework with more data / more specific data accordingly. Although we could use `hyperopt` to tune parameters of a [LDA](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) pipeline, we wanted to explicitly set the number of topics to 6 in the context of that solution accelerator, "lock" that experiment to a define set of topics we could learn more and interpret.

# COMMAND ----------

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words = stop_words, ngram_range = (1,1))
vec_model = vectorizer.fit(corpus)

# COMMAND ----------

import mlflow
from sklearn.pipeline import make_pipeline
from sklearn.decomposition import LatentDirichletAllocation

with mlflow.start_run(run_name='esg_lda') as run:

  lda = LatentDirichletAllocation(
    n_components=6,
    max_iter=150,
    evaluate_every=3,
    random_state=42,
    verbose=False
  )

  # train pipeline
  pipeline = make_pipeline(vec_model, lda)
  pipeline.fit(corpus)

  # log model
  mlflow.sklearn.log_model(pipeline, 'pipeline')

  # Mlflow run ID
  lda_run_id = mlflow.active_run().info.run_id

# COMMAND ----------

# MAGIC %md
# MAGIC ### Interpreting results
# MAGIC We want to evaluate model relevance using more domain expertise. Would those topics make sense from an ESG perspective? Do we have clear categories defined spanning accross the Environmental, Social and Governance broader categories? By interacting with our model through simple visualizations, we want to name each topic into a specific policy in line with [GRI standards](https://www.globalreporting.org/standards/).

# COMMAND ----------

import matplotlib.pyplot as plt
from wordcloud import WordCloud

# We ensure relevance of our topics using simple wordcloud visualisation
def word_cloud(model, tf_feature_names, index):

    imp_words_topic=""
    comp = model.components_[index]
    tfs = ['_'.join(t.split(' ')) for t in tf_feature_names]
    vocab_comp = zip(tfs, comp)
    sorted_words = sorted(vocab_comp, key = lambda x:x[1], reverse=True)[:200]

    for word in sorted_words:
        imp_words_topic = imp_words_topic + " " + word[0]

    return WordCloud(
        background_color="white",
        width=300,
        height=300,
        contour_width=2,
        contour_color='steelblue'
    ).generate(imp_words_topic)

topics = len(lda.components_)
tf_feature_names = vectorizer.get_feature_names()
fig = plt.figure(figsize=(20, 20 * topics / 3))

# Display wordcloud for each extracted topic
for i, topic in enumerate(lda.components_):
    ax = fig.add_subplot(topics, 3, i + 1)
    wordcloud = word_cloud(lda, tf_feature_names, i)
    ax.imshow(wordcloud)
    ax.axis('off')

plt.savefig(f"{temp_directory}/wordcloud.png")

# COMMAND ----------

# MAGIC %md
# MAGIC For different PDF documents, you may have to look at wordcloud visualizations above and rename topics accordingly. As mentioned, different data will yield different insights, and more documents to learn from may result in more specific themes / categories. We leave interpretation of those topics to our users with the ability to map those to internal policies or external guidelines / regulations. We can infer topic distribution for each statement that we store to a delta table. For each statement, we retrieve the most descriptive topic and its meaning given our nomenclature defined above.

# COMMAND ----------

from typing import Iterator
from pyspark.sql.functions import pandas_udf
lda_run_id_B = sc.broadcast(lda_run_id)

@pandas_udf("array<float>")
def describe_topics(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
  import mlflow
  model = mlflow.sklearn.load_model("runs:/{}/pipeline".format(lda_run_id_B.value))
  for batch in batch_iter:
    predictions = model.transform(batch)
    yield pd.Series([[float(p) for p in distribution] for distribution in predictions])

# COMMAND ----------

esg_df \
  .withColumn('probabilities', describe_topics('lemma')) \
  .drop('lemma') \
  .write \
  .format('delta') \
  .mode('overwrite') \
  .option('path', config['csr_statements_path']) \
  .saveAsTable(config['csr_statements_table'])

# COMMAND ----------

display(spark.read.table(config['csr_statements_table']))

# COMMAND ----------

# MAGIC %md
# MAGIC ### Retrieve initiatives
# MAGIC Using a partitioning window, we extract the most descriptive policies for each organization. Although a few policies seems to be properly categorized, we can observe some obvious misclassification. At a first glance, we observe statements containing metrics to be often "recognized" as environmental related. Given the quantifiable nature of environmental policies, our model may be biased towards the use of number, percent signs or metrics. However, despite some misclassification (remember that we used topic modelling to discover unknown themes rather than classifying known labels), we show how one could dramatically simplify a complex PDF document of hundreds of pages into specific initiatives, answering questions like "*What did company X do with regards to environmental policy?*"

# COMMAND ----------

from pyspark.sql.window import Window
import numpy as np

@F.udf('int')
def get_topic(xs):
  return int(np.argmax(xs))

spark \
  .read \
  .table(config['csr_statements_table']) \
  .withColumn('id', get_topic('probabilities')) \
  .withColumn('probability', F.array_max('probabilities')) \
  .withColumn('rank', F.row_number().over(Window.partitionBy('organisation', 'id').orderBy(F.desc('probability')))) \
  .select('organisation','url', 'statement', 'rank', 'probability', 'id') \
  .write \
  .format('delta') \
  .mode('overwrite') \
  .option('path', config['csr_initiatives_path']) \
  .saveAsTable(config['csr_initiatives_table'])

# COMMAND ----------

display(
  spark
    .read
    .table(config['csr_initiatives_table'])
    .withColumnRenamed('id', 'topic_id')
    .filter(F.col('rank') == 1)
    .filter(F.col('probability') > 0.8)
    .orderBy(F.desc('probability'))
    .select('organisation', 'topic_id', 'statement')
)

# COMMAND ----------

# MAGIC %md
# MAGIC As our framework was built around the use of AI, the themes we learned from will be consistent across every organisations. In addition to summarizing complex PDF documents, this ML driven framework can be used to quantify the un-quantifiable and objectively compare non metrics initiatives across organisations, answering questions like "*How much more does company X focus on the wellbeing of their employees compare to company Y?*". For that purpose, we create a simple pivot table that will summarize companies' focus across our machine learned policies.

# COMMAND ----------

esg_group = spark.read.table(config['csr_statements_table']).toPandas()
esg_group['topics'] = esg_group['probabilities'].apply(lambda xs: [[i, x] for i, x in enumerate(xs)])
esg_group = esg_group.explode('topics').reset_index(drop=True)
esg_group['topic_id'] = esg_group['topics'].apply(lambda x: x[0])
esg_group['probability'] = esg_group['topics'].apply(lambda x: x[1])
esg_group = esg_group[['organisation', 'topic_id', 'probability']]
esg_group = pd.pivot_table(
  esg_group,
  values='probability',
  index='organisation',
  columns=['topic_id'],
  aggfunc=np.sum)

# COMMAND ----------

# scale topic frequency between 0 and 1
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range = (0, 1))
esg_focus = pd.DataFrame(scaler.fit_transform(esg_group), columns=esg_group.columns)
esg_focus.index = esg_group.index

# plot heatmap, showing main area of focus for each company across topics we learned
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(rc={'figure.figsize':(12,8)})
sns.heatmap(esg_focus, annot=False, cmap='Greens')
plt.show()

# COMMAND ----------

# MAGIC %md
# MAGIC Finally, we were able to create a framework that helps financial analysts objectively assess the sustainable impact of their investments, retailers to compare the ethical posture of their suppliers or organisations to compare their environmental initiatives with their closest competitors.

# COMMAND ----------

# MAGIC %md
# MAGIC ## CSR score
# MAGIC In the previous section, we set the foundations to a AI driven ESG framework by learning key ESG initiatives as opposition to broad statements. By looking at how descriptive each statement is, we create a simple score by rank ordering organisations. This score will be the building block to our next notebook where we will be able to quantify how much a company talks about ESG vs. how much they walk the talk.

# COMMAND ----------

_ = (
  spark
    .read
    .table(config['csr_statements_table'])
    .withColumn('probabilities', with_topic('probabilities'))
    .withColumn('probabilities', F.explode('probabilities'))
    .select(
      F.col('probabilities.id'),
      F.col('organisation'),
      F.col('probabilities.probability')
    )
    .groupBy('id', 'organisation')
    .agg(
      F.sum('probability').alias('esg')
    )
    .withColumn('rank', F.row_number().over(Window.partitionBy('id').orderBy('esg')))
    .withColumn('esg', F.round(F.col('rank') * 100 / F.lit(esg_pdf.shape[0])))
    .withColumnRenamed('id', 'topic_id')
    .select('organisation', 'topic_id', 'esg')
    .write
    .format('delta')
    .mode('overwrite')
    .option('path', config['csr_scores_path'])
    .saveAsTable(config['csr_scores_table'])
)

# COMMAND ----------

esg_csr_data = (
  spark
    .read
    .table(config['csr_scores_table'])
    .withColumnRenamed('id', 'topic_id')
    .groupBy('organisation')
    .agg(F.avg('esg').alias('esg'))
    .toPandas()
)

esg_csr_data['sum'] = esg_csr_data["esg"].sum()
esg_csr_data.index = esg_csr_data['organisation']
esg_csr_data = esg_csr_data.sort_values(by='sum', ascending=False).drop('sum',  axis=1)
esg_csr_data.plot.bar(
  rot=90,
  stacked=False,
  title='ESG score based on corporate disclosures',
  ylabel='ESG score',
  ylim=[0, 100],
  figsize=(16, 8)
)

# COMMAND ----------

# MAGIC %md
# MAGIC ## Actionable models
# MAGIC Now that we've built our model, we may want to port it out to classify different documents or transfer the intelligence we learned to a different data feed such as news articles. However, we built a few pipelines and text preparation (such as lemmatization) that would need to be shipped alongside the LDA model itself. This data preparation can be embedded as part of a `pyfunc` model as follows. Our approach is to wrap our lemmatization process as data preparation for topic classification and ensure each python dependency required will be explicitly embedded with the relevant version to run independently.

# COMMAND ----------

class EsgTopicAPI(mlflow.pyfunc.PythonModel):

  def __init__(self, pipeline):
    self.pipeline = pipeline

  def load_context(self, context):
    import nltk
    nltk.data.path.append(context.artifacts['wordnet'])

  def _lemmatize(self, text):
    import nltk
    from nltk.stem import WordNetLemmatizer, PorterStemmer
    from gensim.utils import simple_preprocess
    lemmatizer = WordNetLemmatizer()
    stemmer = PorterStemmer()
    results = [stemmer.stem(lemmatizer.lemmatize(token)) for token in simple_preprocess(text)]
    return ' '.join(results)

  def predict(self, context, series):
    lemma = series.apply(self._lemmatize)
    predictions = pipeline.transform(lemma)
    import pandas as pd
    return pd.Series([[float(p) for p in distribution] for distribution in predictions])

# COMMAND ----------

import sklearn

with mlflow.start_run(run_name='esg_topic_classification'):

  conda_env = mlflow.pyfunc.get_default_conda_env()
  conda_env['dependencies'][2]['pip'] += ['scikit-learn=={}'.format(sklearn.__version__)]
  conda_env['dependencies'][2]['pip'] += ['gensim=={}'.format(gensim.__version__)]
  conda_env['dependencies'][2]['pip'] += ['nltk=={}'.format(nltk.__version__)]
  conda_env['dependencies'][2]['pip'] += ['pandas=={}'.format(pd.__version__)]
  conda_env['dependencies'][2]['pip'] += ['numpy=={}'.format(np.__version__)]

  wordnet_dir = "/dbfs{}/wordnet".format(data_path)
  nltk.download('wordnet', download_dir=wordnet_dir)
  artifacts = {
    'wordnet': wordnet_dir
  }

  mlflow.pyfunc.log_model(
    'pipeline',
    python_model=EsgTopicAPI(pipeline),
    conda_env=conda_env,
    artifacts=artifacts
  )

  api_run_id = mlflow.active_run().info.run_id
  print(api_run_id)

# COMMAND ----------

# MAGIC %md
# MAGIC By registering our model to ML registry, we make it available to downstream processes and backend jobs. We will simply load our `pyfunc` logic as a simple mlflow import.

# COMMAND ----------

client = mlflow.tracking.MlflowClient()
model_uri = "runs:/{}/pipeline".format(api_run_id)
result = mlflow.register_model(model_uri, config['model_topic_name'])
version = result.version

# COMMAND ----------

# MAGIC %md
# MAGIC We can also promote our model to different stages programmatically. Although our models would need to be reviewed in real life scenario, we make it available as a production artifact for our next notebook focused on news analytics as well as archiving any previous iteration

# COMMAND ----------

client = mlflow.tracking.MlflowClient()
for model in client.search_model_versions("name='{}'".format(config['model_topic_name'])):
  if model.current_stage == 'Production':
    print("Archiving model version {}".format(model.version))
    client.transition_model_version_stage(
      name=config['model_topic_name'],
      version=int(model.version),
      stage="Archived"
    )

# COMMAND ----------

client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
    name=config['model_topic_name'],
    version=version,
    stage="Production"
)

# COMMAND ----------

# MAGIC %md
# MAGIC ## Take away
# MAGIC In this first section, we set the foundations to a data driven ESG framework. We've demonstrated how AI can be used to extract key ESG initiatives from unstructured PDF documents and use this intelligence to create a more objective way to quantify ESG strategies from public companies. With the vocabulary we have learned using topic modelling, we can re-use that model to see how much of these initiatives were actually followed through and what was the media reception using news analytics data.

{'nltk': 'nltk', 'WordNetLemmatizer': 'nltk.stem.WordNetLemmatizer', 'PorterStemmer': 'nltk.stem.PorterStemmer', 'simple_preprocess': 'gensim.utils.simple_preprocess', 'pd': 'pandas'}


import os
import sys
import time

import numpy as np

import theano
import theano.tensor as T
from theano.tensor.shared_randomstreams import RandomStreams

import PIL.Image as Image

from collections import OrderedDict

def stochasticGradient(cost,params,lr = 0.01):
    '''
    Stochastic Gradient Descent
    '''
    grads = T.grad(cost, params)
    updates = OrderedDict()

    for param, grad in zip(params, grads):
        updates[param] = param - lr * grad

    return updates

def apply_momentum(updates, params=None, momentum=0.9):
    """
    lasagne
    """

    if params is None:
        params = updates.keys()
    updates = OrderedDict(updates)

    for param in params:
        value = param.get_value(borrow=True)
        velocity = theano.shared(np.zeros(value.shape, dtype=value.dtype),
                                    broadcastable=param.broadcastable)
        x = momentum * velocity + updates[param]
        updates[velocity] = x - param
        updates[param] = x

    return updates

def momentum(cost, params, lr = 0.01, momentum=0.9):
    """
    lasagne
    """
    updates = stochasticGradient(cost, params, lr = lr)
    return apply_momentum(updates, momentum=momentum)

class dA(object):

    def __init__(
        self,
        numpy_rng,
        theano_rng=None,
        input=None,
        n_visible=784,
        n_hidden=500,
        W=None,
        bhid=None,
        bvis=None
    ):
        self.n_visible = n_visible
        self.n_hidden = n_hidden

        if not theano_rng:
            theano_rng = RandomStreams(np_rng.randint(2 ** 30))

        if not W:
            initial_W = np.asarray(
                np_rng.uniform(
                    low=- np.sqrt(2. / (n_hidden + n_visible)),
                    high= np.sqrt(2. / (n_hidden + n_visible)),
                    size=(n_visible, n_hidden)
                ),
                dtype=theano.config.floatX
            )
            W = theano.shared(value=initial_W, name='W', borrow=True)

        if not bvis:
            bvis = theano.shared(
                value=np.zeros(
                    n_visible,
                    dtype=theano.config.floatX
                ),
                borrow=True
            )

        if not bhid:
            bhid = theano.shared(
                value=np.zeros(
                    n_hidden,
                    dtype=theano.config.floatX
                ),
                name='b',
                borrow=True
            )

        self.W = W
        self.b = bhid
        self.b_prime = bvis
        self.W_prime = self.W.T
        self.theano_rng = theano_rng
        self.x = input
        self.params = [self.W, self.b, self.b_prime]

    def get_corrupted_input(self, input, corruption_level):
        return self.theano_rng.binomial(size=input.shape, n=1,
                                        p=1 - corruption_level,
                                        dtype=theano.config.floatX) * input

    def rectify(self,X):
        return T.maximum(0,X)

    def get_hidden_values(self, input):
        return self.rectify(T.dot(input, self.W) + self.b)

    def get_reconstructed_input(self, hidden):
        return T.dot(hidden, self.W_prime) + self.b_prime

    def get_cost_updates(self, corruption_level, learning_rate):
        """ This function computes the cost and the updates for one trainng
        step of the dA """

        tilde_x = self.get_corrupted_input(self.x, corruption_level)
        y = self.get_hidden_values(tilde_x)
        z = self.get_reconstructed_input(y)

        cost = T.mean((self.x-z)**2)

        #gparams = T.grad(cost, self.params)
        #updates = [
        #    (param, param - learning_rate * gparam)
        #    for param, gparam in zip(self.params, gparams)
        #]

        updates = stochasticGradient(cost, self.params,lr=learning_rate)

        return (cost, updates, y, self.get_reconstructed_input)

if __name__ == '__main__':
    pass


# from urllib import request

from flask import Flask, render_template,request,make_response
from sklearn.pipeline import Pipeline
import joblib
from sklearn import linear_model
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
import feature as fp
import PyPDF2
app = Flask(__name__)
loaded_model = joblib.load('news.sav')
def download_data(num):

    if num == 2:
        response = make_response(open('news_2.pdf', 'rb').read())

        return response
    # if num == 3:
    #     response = make_response(open('news_3.pdf', 'rb').read())
    #     response.headers['Content-Type'] = 'pdf'
    #     response.headers["Content-Disposition"] = "attachment; filename=data-3.pdf"
    #     return response
    # if num == '4':
    #     response = make_response(open('news_4.pdf', 'rb').read())
    #     response.headers['Content-Type'] = 'pdf'
    #     response.headers["Content-Disposition"] = "attachment; filename=data-4.pdf"
    #     return response

@app.route('/')
def index():  # put application's code here
    return render_template('index.html')
@app.route("/upload",methods=["POST"])
def upload():
    if request.method == 'POST':
        file_obj = request.files['upload']
        pdfReader = PyPDF2.PdfFileReader(file_obj)

        pageObj = pdfReader.getPage(0)

        sentence = (pageObj.extractText())
            # print(sentence)
        result= fp.manual_testing(sentence)
    # if request.form['news1'] == '1':
    #     download(1)
    return render_template("sucess.html", result=result)
@app.route("/download",methods=["POST"])
def download():
    if request.method == 'POST':
        data1=request.form.get('news1')
        data2=request.form.get('news2')
        data3=request.form.get('news3')
        data4=request.form.get('news4')
        if(data1=='1'):
            response = make_response(open('news_1.pdf', 'rb').read())
            response.headers['Content-Type'] = 'pdf'
            response.headers["Content-Disposition"] = "attachment; filename=data-1.pdf"
            return response
        if(data2=='2'):
            response = make_response(open('news_2.pdf', 'rb').read())
            response.headers['Content-Type'] = 'pdf'
            response.headers["Content-Disposition"] = "attachment; filename=data-2.pdf"
            return response
        if(data3=='3'):
            response = make_response(open('news_3.pdf', 'rb').read())
            response.headers['Content-Type'] = 'pdf'
            response.headers["Content-Disposition"] = "attachment; filename=data-3.pdf"
            return response
        if(data4=='4'):
            response = make_response(open('news_4.pdf', 'rb').read())
            response.headers['Content-Type'] = 'pdf'
            response.headers["Content-Disposition"] = "attachment; filename=data-4.pdf"
            return response

if __name__ == '__main__':
    app.run()


import os
import subprocess
import sys
import logging
import shutil
from flask import Flask, jsonify, render_template, request
from werkzeug import secure_filename

app = Flask(__name__)
app.logger.addHandler(logging.StreamHandler(sys.stdout))
app.logger.setLevel(logging.ERROR)
app.config['PROPAGATE_EXCEPTIONS'] = True
app.config['TEMP_FOLDER'] = '/tmp'
app.config['OCR_OUTPUT_FILE'] = 'ocr'
app.config['MAX_CONTENT_LENGTH'] = 4 * 1024 * 1024

def allowed_file(filename):
  return '.' in filename and filename.rsplit('.', 1)[1] in set(['png', 'jpg', 'jpeg', 'gif', 'tif', 'tiff'])

@app.errorhandler(404)
def not_found(error):
  resp = jsonify( {
    u'status': 404,
    u'message': u'Resource not found'
  } )
  resp.status_code = 404
  return resp

@app.route('/')
def api_root():
  resp = jsonify( {
    u'status': 200,
    u'message': u'Welcome to our secret APIs'
  } )
  resp.status_code = 200
  return resp

@app.route('/test', methods = ['GET'])
def test():
  return render_template('upload_form.html', landing_page = 'process')

@app.route('/process', methods = ['GET','POST'])
def process():
  if request.method == 'POST':
    file = request.files['file']
    hocr = request.form.get('hocr') or ''
    ext = '.hocr' if hocr else '.txt'
    if file and allowed_file(file.filename):
      folder = os.path.join(app.config['TEMP_FOLDER'], str(os.getpid()))
      os.mkdir(folder)
      input_file = os.path.join(folder, secure_filename(file.filename))
      output_file = os.path.join(folder, app.config['OCR_OUTPUT_FILE'])
      file.save(input_file)

      command = ['tesseract', input_file, output_file, '-l', request.form['lang'], hocr]
      proc = subprocess.Popen(command, stderr=subprocess.PIPE)
      proc.wait()

      output_file += ext

      if os.path.isfile(output_file):
        f = open(output_file)
        resp = jsonify( {
          u'status': 200,
          u'ocr':{k:v.decode('utf-8') for k,v in enumerate(f.read().splitlines())}
        } )
      else:
        resp = jsonify( {
          u'status': 422,
          u'message': u'Unprocessable Entity'
        } )
        resp.status_code = 422

      shutil.rmtree(folder)
      return resp
    else:
      resp = jsonify( {
        u'status': 415,
        u'message': u'Unsupported Media Type'
      } )
      resp.status_code = 415
      return resp
  else:
    resp = jsonify( {
      u'status': 405,
      u'message': u'The method is not allowed for the requested URL'
    } )
    resp.status_code = 405
    return resp

if __name__ == '__main__':
  app.run(debug=True)


from accounts.forms.profile_form import ProfileForm
from accounts.models import Profile
from django.contrib.auth.decorators import login_required
from django.db.models import Q
from django.http import Http404
from django.shortcuts import redirect, render
from django.urls import reverse, reverse_lazy
from django.utils.decorators import method_decorator
from django.views import View
from django.views.generic import DetailView, ListView, UpdateView

from .models import Friend

@method_decorator(login_required, 'dispatch')
class IndexView(View):
    def get(self, *args, **kwargs):
        user = self.request.user
        # friends_1 = Friend.objects.filter(status='A', id_requester__user=user).order_by('-id')
        # friends_2 = Friend.objects.filter(status='A', id_receiver__user=user).order_by('-id')
        friends = Friend.objects.filter(Q(Q(id_requester__user=user) | Q(id_receiver__user=user)), status='A').order_by('-id')

        return render(self.request, 'friendships/index.html', {'friends': friends})

@method_decorator(login_required, 'dispatch')
class SearchListView(ListView):
    model = Profile
    context_object_name = 'profiles'
    template_name = 'friendships/search.html'
    paginate_by = 5
    ordering = ['-id']
    def get_queryset(self, *args, **kwargs):
        qs = super().get_queryset(*args, **kwargs)

        surname_search = self.request.GET.get('surname', '')
        profile_user = Profile.objects.get(user=self.request.user)
        qs = qs.filter(
            surname__icontains=surname_search
        ).exclude(surname=profile_user.surname)

        return qs

@method_decorator(login_required, 'dispatch')
class ProfileDetailView(DetailView):
    model = Profile
    context_object_name = 'profile'
    template_name = 'friendships/profile_detail.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # Get profile user logged
        profile_user = Profile.objects.get(user=self.request.user)

        # Slug profil detail
        slug = self.kwargs.get('slug')

        # Get profile from slug
        profile_detail = Profile.objects.get(slug=slug)

        # Checks if slug is equal slug from profile_user
        is_owner = True if slug == profile_user.slug else False
        context['is_owner'] = is_owner

        # Check if are friends
        are_friends = Friend.objects.filter(
            Q(id_requester=profile_user,
            id_receiver=profile_detail) |
            Q(id_requester=profile_detail,
            id_receiver=profile_user)
        ).exists()

        context['are_friends'] = are_friends
        return context

@method_decorator(login_required, 'dispatch')
class ProfileEditUpdateView(UpdateView):
    model = Profile
    form_class = ProfileForm
    template_name = 'friendships/profile-edit.html'
    success_url = reverse_lazy('friendships:index')

    def get(self, request, *args, **kwargs):
        slug = self.kwargs.get('slug')
        profile = Profile.objects.filter(slug=slug)

        if not profile:
            raise Http404()

        if profile.first().user != self.request.user:
            return redirect(reverse('friendships:profile-detail', kwargs={'slug': slug}))

        return super().get(request, *args, **kwargs)

@method_decorator(login_required, 'dispatch')
class ProfileFriendRequest(View):
    def get(self, *args, **kwargs):
        slug = self.kwargs.get('slug')

        profile_request = Profile.objects.filter(slug=slug).first()

        if not profile_request:
            raise Http404()

        profile_owner = Profile.objects.get(user=self.request.user)

        if profile_request.slug == profile_owner.slug:
            return redirect(reverse_lazy('friendships:index'))

        search_friendships = Friend.objects.filter(
            Q(
                Q(id_requester = profile_request,
                id_receiver = profile_owner),
                ~Q(status = 'D')
            ) | Q(
                Q(id_requester = profile_owner,
                id_receiver = profile_request),
                ~Q(status = 'D')
            )
        ).exists()

        if search_friendships:
            return redirect(reverse_lazy('friendships:index'))

        Friend.objects.create(id_requester=profile_owner, id_receiver=profile_request)
        return redirect(reverse_lazy('friendships:index'))

@method_decorator(login_required, 'dispatch')
class FriendsPendingView(View):
    def get(self, *args, **kwargs):
        user = self.request.user
        friends = Friend.objects.filter(Q(id_requester__user=user), status='P').order_by('-id')

        return render(self.request, 'friendships/pending.html', {'friends': friends})

@method_decorator(login_required, 'dispatch')
class DeniedRequestView(View):
    def get(self, *args, **kwargs):
        user = self.request.user

        id_url = self.kwargs.get('id')
        friend_request = Friend.objects.filter(
            id_requester__user=user,
            id=id_url,
            status='P',
        ).first()

        if not friend_request:
            return redirect(reverse('friendships:pending'))

        friend_request.status = 'D'
        friend_request.save()

        return redirect(reverse('friendships:pending'))

@method_decorator(login_required, 'dispatch')
class AcceptedRequestView(View):
    def get(self, *args, **kwargs):
        user = self.request.user

        id_url = self.kwargs.get('id')
        friend_request = Friend.objects.filter(
            id_requester__user=user,
            id=id_url,
            status='P',
        ).first()

        if not friend_request:
            return redirect(reverse('friendships:pending'))

        friend_request.status = 'A'
        friend_request.save()

        return redirect(reverse('friendships:pending'))


import gspread
from oauth2client.service_account import ServiceAccountCredentials
import os
import time

time_start=time.clock()

scope = [r'https://spreadsheets.google.com/feeds']

#os.chdir('discord_bots\\pss_bot')

credentials = ServiceAccountCredentials.from_json_keyfile_name('pss_bot.json', scope)

gc = gspread.authorize(credentials)

sheet = gc.open_by_key('1nJFO_nksOxlAW4VMgUjrqljOjKv8CKN665M72h3Jspw')

prestiges = {} #Dictionary containing all prestige information

legendary = sheet.worksheet("Legendary-PWW link").get_all_values() #singular indicates sheet name
hero = sheet.worksheet("Heroes-PWW link").get_all_values()
epic = sheet.worksheet("Epics-PWW link").get_all_values()

def all_names(sheet): #Assembles all names in the sheet
    return [x for x in sheet[0] if x!='']

def add_to_prestiges(name_list, source_sheet): #Adds all values in that sheet to the Prestige dictionary
    for name1 in range(1,len(name_list)+1):
        for name2 in range(1,len(name_list)+1):
            prestiges["%s_%s"%(name_list[name1-1],name_list[name2-1])]=source_sheet[name1][name2]

legendaries = all_names(legendary) #plural indicates name list
heroes = all_names(hero)
epics = all_names(epic)

add_to_prestiges(legendaries,legendary)
add_to_prestiges(heroes,hero)
add_to_prestiges(epics,epic)

print ('Data prep complete. Time Elapsed :'+str(time.clock()-time_start))

import discord
from discord.ext import commands
import random

description = """A bot designed to relay known information regarding Pixel Starships."""

bot = commands.Bot(command_prefix='?', description=description)

@bot.event
async def on_ready():
    print('Logged in as')
    print(bot.user.name)
    print(bot.user.id)
    print('------')

@bot.command()
async def namelist(request : str):
    """returns a list of valid names"""
    if request.lower()=='legendary' or request.lower()=='legendaries':
        await bot.say('The following is a list of all known names of Legendary Crew. I will likely only understand names that are typed as follows:')
        await bot.say(legendaries)
    elif request.lower()=='hero' or request.lower()=='heroes':
        await bot.say('The following is a list of all known names of Hero Crew. I will likely only understand names that are typed as follows:')
        await bot.say(heroes)
    elif request.lower()=='epic' or request.lower()=='epics':
        await bot.say('The following is a list of all known names of Epic Crew. I will likely only understand names that are typed as follows:')
        await bot.say(epics)
    else:
        await bot.say('Valid inputs include legendary, hero, or epic')

@bot.command()
async def prestige(*,request : str):
    """Returns data on prestige results for the two crew members entered as input"""
    names = request.title().split(',')
    if names[1][0] == ' ':
        names[1] = names[1][1:]
    try:
        await bot.say(prestiges["%s_%s"%(names[0],names[1])])
    except:
        await bot.say("I didn't quite catch that. Make sure the two crew names are separated by a comma (,) and that they are spelled correctly. Reference ?namelist for names I recognize.")
    finally:
        if prestiges["%s_%s"%(names[0],names[1])]=='':
            await bot.say("I don't appear to have any results for this combination.")

bot.run('Mjc0NDU1NTg5NTY2ODA4MDc2.C2yWMw.uFEcuSxiZRJPp-UkloTvWV0z84Q')


# coding=utf-8

import os
import sys
import time
import tkinter
from tkinter import messagebox

import requests

from myConvert import ConvertPDF, ConvertPic

class tkPDF:
    """
    创建PDF转换的Tk窗体\n
    width: 窗体宽度\n
    height: 窗体高度\n
    """

    __Url: str
    __APP_ID: str
    __API_KEY: str
    __SECRET_KEY: str

    window = tkinter.Tk()
    __title: str
    __width: int
    __height: int

    # 创建Button控件, command: 绑定指定函数
    btn_pic = tkinter.Button(window,
                             font=('宋体', 11, 'normal'),
                             text='图片识别文字提取')
    btn_pdf_img = tkinter.Button(window,
                                 font=('宋体', 11, 'normal'),
                                 text='PDF(扫描版)转WORD文档')
    btn_pdf_word = tkinter.Button(window,
                                  font=('宋体', 11, 'normal'),
                                  text='PDF(文字版)转word文档')
    btn_refresh = tkinter.Button(window, font=('黑体', 9, 'bold'), text='再次刷新')

    # 创建Label控件
    label = tkinter.Label(window,
                          font=('黑体', 11, 'bold'),
                          justify='left',
                          text='今日的高精度识别次数：---\n\n今日的标准识别次数：-----')

    def __init__(self, title, width: int, height: int):
        # 读取配置文件
        self.get_env()

        self.__title = title
        self.__width = width
        self.__height = height

        self.window.title(self.__title)
        self.window.geometry('{0}x{1}'.format(self.__width, self.__height))

        # 绑定函数
        self.btn_pic.config(command=self.On_parse_pic)
        self.btn_pdf_img.config(command=self.On_parse_img)
        self.btn_pdf_word.config(command=self.On_parse_word)
        self.btn_refresh.config(command=self.get_freeCount)

        # 设置width, height
        self.btn_pic.place(width=150, height=40)
        self.btn_pdf_img.place(width=180, height=40)
        self.btn_pdf_word.place(width=180, height=40)
        self.btn_refresh.place(width=100, height=30)

        # 设置Label参数
        self.label.place(x=350, y=220)

        # 设置控件在窗体的位置
        _x = 60  # 左上角x
        _y = 100  # 左上角y
        margin = 30  # 控件之间的间距

        self.btn_pic.place(x=_x, y=_y)
        _x += int(self.btn_pic.place_info()['width'])  # 获取控件的width
        _x += margin

        self.btn_pdf_img.place(x=_x, y=_y)
        _x += int(self.btn_pdf_img.place_info()['width'])  # 获取控件的width
        _x += margin

        self.btn_pdf_word.place(x=_x, y=_y)
        self.btn_refresh.place(x=580, y=228)

        # 获取今日的免费次数
        self.get_freeCount()

        # 开始捕获窗体消息(动态更新)
        self.window.mainloop()

    def get_env(self):
        """
        设置配置文件，添加配置参数
        """
        try:
            INI_file = os.path.join(os.path.dirname(sys.argv[0]), r'app.ini')  # 读取主程序目录下的app.ini
            INI_read = open(INI_file, 'r').read()
            INI_rows = INI_read.split('\n')
            for i in range(len(INI_rows)):
                if INI_rows[i].find('[SERVER]') > -1:
                    self.__Url = INI_rows[i + 1].split('=')[1]  # 获取url值
                elif INI_rows[i].find('[APP_INFO]') > -1:
                    self.__APP_ID = INI_rows[i + 1].split('=')[1]  # 获取app_id值
                    self.__API_KEY = INI_rows[i +
                                              2].split('=')[1]  # 获取api_key值
                    self.__SECRET_KEY = INI_rows[i + 3].split('=')[
                        1]  # 获取secret_key值
        except BaseException:
            messagebox.showinfo('提示', '未能加载配置文件: app.ini ，请检查文件是否存在。')
            sys.exit(0)

    def get_freeCount(self):
        """
        获取今日的免费次数
        """

        url_params = {'a': self.__APP_ID, 't': int(time.time())}
        res = requests.get(url=self.__Url, params=url_params)
        if res.status_code == 200:
            result = res.json()
            if result['errCode'] == 0:
                self.label.config(text='今日的高精度识别次数：{}\n\n今日的标准识别次数：{}'.format(
                    500 - int(result['count']['bA']), 50000 -
                    int(result['count']['bG'])))
                self.active_btn()
            else:
                messagebox.showinfo('提示', '获取今天次数失败，请检查网络。')
                self.disable_btn()
        else:
            messagebox.showinfo('提示', '网络中断，请检查网络。')
            self.disable_btn()

    def getInfo_app(self):
        """
        获取AipOcr的参数列表
        """

        return [self.__APP_ID, self.__API_KEY, self.__SECRET_KEY]

    def disable_btn(self):
        """
        禁用按钮
        """

        self.btn_refresh.config(state='active')
        self.btn_pic.config(state='disabled')
        self.btn_pdf_img.config(state='disabled')
        self.btn_pdf_word.config(state='disabled')

    def active_btn(self):
        """
        激活按钮
        """

        self.btn_refresh.config(state='disabled')
        self.btn_pic.config(state='active')
        self.btn_pdf_img.config(state='active')
        self.btn_pdf_word.config(state='active')

    def On_parse_pic(self):
        """
        开始图片文字识别
        """

        # 创建转换Pic的实例类
        mypic = ConvertPic(self.getInfo_app(), self.__Url)
        if mypic.errCode == 0:
            self.window.title('正在转换中，请等待...')
            callback = mypic.mult_parse_pic()
            if callback['errCode'] == 0:
                self.window.title('PDF转换 功能选择')
                messagebox.showinfo('提示', callback['message'])
                self.label.config(text='今日的高精度识别次数：{}\n\n今日的标准识别次数：{}'.format(
                    500 - int(callback['count']['bA']), 50000 -
                    int(callback['count']['bG'])))
                os.startfile(mypic.filedir)
            elif callback['errCode'] == 210:
                messagebox.showinfo('提示', '今天的次数已用完，请等明天。')
            else:
                messagebox.showinfo('提示', '网络中断，请检查网络。')
        else:
            messagebox.showinfo('提示', mypic.errMessage)

    def On_parse_img(self):
        """
        开始PDF的图片文字识别
        """

        # 创建转换PDF的实例类
        mypdf = ConvertPDF(self.getInfo_app(), self.__Url)
        if mypdf.errCode == 0:
            self.window.title('正在转换中，请等待...')
            callback = mypdf.mult_parse_img()
            if callback['errCode'] == 0:
                self.window.title('PDF转换 功能选择')
                messagebox.showinfo('提示', callback['message'])
                self.label.config(text='今日的高精度识别次数：{}\n\n今日的标准识别次数：{}'.format(
                    500 - int(callback['count']['bA']), 50000 -
                    int(callback['count']['bG'])))
                os.startfile(mypdf.filedir)
            elif callback['errCode'] == 210:
                messagebox.showinfo('提示', '今天的次数已用完，请等明天。')
            else:
                messagebox.showinfo('提示', '网络中断，请检查网络。')
        else:
            messagebox.showinfo('提示', mypdf.errMessage)

    def On_parse_word(self):
        """
        开始PDF的文字识别
        """

        # 创建转换PDF的实例类
        mypdf = ConvertPDF()
        if mypdf.errCode == 0:
            self.window.title('正在转换中，请等待...')
            callback = mypdf.mult_parse_word()
            self.window.title('PDF转换 功能选择')
            messagebox.showinfo('提示', callback['message'])
            os.startfile(mypdf.filedir)
        else:
            messagebox.showinfo('提示', mypdf.errMessage)

if __name__ == '__main__':
    myTk = tkPDF("PDF转换 功能选择", 700, 300)


# to capture images from webcam
import cv2
# to work on arrays
import numpy as np

# opens webcam and gives us the handle back
capture = cv2.VideoCapture(0)
# load face detection classifier
dataset = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")

# an empty to store images later on
data = []

# an infinite loop to keep the webcam on
while True:
    # starts reading data (capture image) from webcam
    ret, img = capture.read()
    # ret tells us the status of webcam -> true means webcam is on and running fine
    if ret:
        # convert image to grayscale as grayscale image is lighter (2D array, not 3D)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        # detect faces in the image captured from webcam and zoom image by 30% to improve face detection
        faces = dataset.detectMultiScale(gray, 1.3)
        # put a loop and extract the details related to each face detected in the image
        # x,y -> top-left coords of face rectangle; w,h -> width and height of face rectangle
        for x,y,w,h in faces:
            # draw a rectangle on the image to show where the face was detected
            # colored_image, top-left coords of rect, bottom-right coords of rect, color of rect, border width of rect
            cv2.rectangle(img, (x,y), (x+w, y+h), (255,0,0), 5)
            # slice the face from image so we can store only the face not the complete image
            # slice rows first, then columns, then color channel (bgr)
            myFace = img[y:y+h, x:x+w, : ]
            # myFace will be a nested list of rgb values which is not going to be of same size always
            # so we need to resize all the lists for each face to a specified size and make comparisons error free
            myFace = cv2.resize(myFace, (50,50))
            # keep storing the faces in data list until you have got 100 faces
            if len(data) < 100:
                # append the face list to the data list
                data.append(myFace)
                # print length of data list to keep a check of how many faces have been captured
                print(len(data))
        # show image so that we can see where the face is detected
        cv2.imshow('result', img)
        # check for any events related to esc key or break the while loop if you have captured 100 images
        if cv2.waitKey(1) & 0xFF == 27 or len(data) == 100:
            break
    # if camera doesn't start because of some issue
    else:
        print("Camera not working")
        break

# convert list to array so that it can be stored easily and take very small storage space
data = np.array(data)
# store the array in the system
np.save('lucifer.npy', data)
# turn off the webcam
capture.release()
# close all the windows opened by cv2
cv2.destroyAllWindows()

'''
data.shape
(100, 50, 50, 3)
>>> import matplotlib.pyplot as plt
>>> data[0]
face1 = data[0]
>>> plt.imshow(face1)
<matplotlib.image.AxesImage object at 0x120aa4898>
>>> plt.show()
>>> face1 = data[50]
>>> plt.imshow(face1)
<matplotlib.image.AxesImage object at 0x120dc3748>
>>> plt.show()
'''


# -*- coding: utf-8 -*-
"""currency view."""
from flask import Blueprint, jsonify, request
from flask.views import MethodView
from flask_jwt_extended import verify_jwt_in_request
from marshmallow import ValidationError
from sqlalchemy import func

from evolux_solution.database import db
from evolux_solution.views import ViewPaginator

from .models import Currency
from .serializer import CurrencySchema

blueprint = Blueprint("currency", __name__, url_prefix="/api/currencies")

class CurrencyAPI(MethodView, ViewPaginator):
    """Currency API CRUD."""

    def __init__(self):
        """Checks access authorization in the requests."""
        verify_jwt_in_request()

    def get(self, code=None):
        """Get currencies availables or filter by code."""
        currency_schema = CurrencySchema()
        if code is None:
            return self.get_paginated_response(Currency.query, CurrencySchema)
        else:
            currency = Currency.query.filter(
                func.lower(Currency.code) == func.lower(code)
            ).first()
            if currency is None:
                return jsonify({"message": "Currency not found"}), 400
            return jsonify(currency_schema.dump(currency))

    def post(self):
        """Create new currency record."""
        try:
            json_data = request.json
            if not json_data:
                return jsonify({"message": "Invalid request"}), 400

            currency_schema = CurrencySchema()
            currency = currency_schema.load(json_data, session=db.session)
            currency.save()
            return jsonify(currency_schema.dump(currency)), 201
        except ValidationError as err:
            return jsonify({"errors": err.messages}), 400

    def put(self, code):
        """Update currency record."""
        try:
            json_data = request.json
            if not json_data:
                return jsonify({"message": "Invalid request"}), 400

            currency_schema = CurrencySchema()
            currency = Currency.query.filter(
                func.lower(Currency.code) == func.lower(code)
            ).first()
            if currency is None:
                return jsonify({"message": "Currency not found"}), 400

            new_currency = currency_schema.load(json_data, instance=currency)
            new_currency.save()
            return jsonify(currency_schema.dump(currency)), 200
        except ValidationError as err:
            return jsonify({"errors": err.messages}), 400

    def delete(self, code):
        """Delete currency record."""
        currency = Currency.query.filter(
            func.lower(Currency.code) == func.lower(code)
        ).first()
        if currency is None:
            return jsonify({"message": "Currency not found."}), 400

        currency.delete()
        return jsonify({"message": "Currency deleted."}), 200

currency_view = CurrencyAPI.as_view("currencies")
blueprint.add_url_rule(
    "/",
    view_func=currency_view,
    methods=["GET", "POST"],
)
blueprint.add_url_rule(
    "/<string:code>",
    view_func=currency_view,
    methods=[
        "GET",
        "PUT",
        "DELETE",
    ],
)


import pygame
from pygame.locals import *
import FrontEnd as f
import BackEnd as b
import Fight

WHITE = 250,250,250

#This is where the main loops are... main game control

# game class that is instantiated in the Launcher
class Game():

	# Initialise what we need
	def __init__(self, screenSize, name):
		self.screen = f.Screen(screenSize, name)
		# A Clock is needed to limit our framerate
		self.clock = pygame.time.Clock()
		# Options loaded in
		self.options = f.OptionConfigs()

		self.fightOptions = None

	# mainloop that controls the flow of the whole game
	def mainLoop(self):

		# Make sure it is possible to get out of the game.
		while True:
			for event in pygame.event.get():
				if event.type == QUIT:
					quit()

			self.menuLoop()

			self.gameLoop()

			self.resultsLoop()

	# menuLoop that controls the flow of the menus
	def menuLoop(self):

		print("Entering Menu Loop")

		menu = b.Pages()
		position = 0
		play = False

		# read the event queue for key presses.
		while True:
			#options = menu.currentPage.connectedPageNameStrings
			numberOptions = len(menu.currentPage.connectedPages) - 1 # used to constrain the selection

			for event in pygame.event.get():
				if event.type == QUIT:
					quit()
				elif event.type == KEYDOWN:
					if event.key == K_UP:
						menu.currentPage.connectedPages[position].highlightOff()
						position = position - 1
						if position < 0: # Ensure that it can't go out of bounds
							position = 0
					elif event.key == K_DOWN:
						menu.currentPage.connectedPages[position].highlightOff()
						position = position + 1
						if position > numberOptions: # ensure it doesn't go below the list
							position = numberOptions
					elif event.key == K_LEFT:
						if menu.currentPage.connectedFrom:
							menu.currentPage = menu.currentPage.connectedFrom
							position = 0
					elif event.key == K_RETURN:
						menu.currentPage.connectedPages[position].highlightOff()
						menu.currentPage = menu.currentPage.connectedPages[position]
						position = 0
					elif event.key == K_ESCAPE:
						play = True

			if play:
				self.fightOptions = "Yay"
				print("Leaving Menu Loop")
				return # A neat way of moving into the game. (might decide to send a list of config data instead)

			menu.currentPage.connectedPages[position].highlightOn()
			self.screen.refreshMenu(menu.currentPage)

	def gameLoop(self):

		print("Entering Game Loop")

		self.screen.whiteout() # Clear the screen
		dead = False
		while not dead:
			dead = Fight.Game()

		print("Leaving Game Loop")

	def resultsLoop(self):

		print("Entering Results Loop")

		self.screen.setTitleText("ResultsLoop")

		while True:
			for event in pygame.event.get():
				if event.type == QUIT:
					quit()
				elif event.type == KEYDOWN:
					print("MenuDOWN")
					if event.key == K_LEFT:
						print("Left")
					elif event.key == K_RIGHT:
						print("Right")
					elif event.key == K_ESCAPE:
						print("Laaving Results Loop")
						return
				elif event.type == KEYUP:
					print("MenuUP")

			self.screen.refreshText()


from django.shortcuts import render
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt

# Create your views here.

import json

from .models import Idea

def index(request):
    idea = Idea.objects.all()
    response = json.dumps([{}])
    return HttpResponse(response, content_type='text/json')

def get_idea(request, idea_name):
    if request.method == "GET":
        try:
            idea = Idea.objects.get(title=idea_name)
            response = json.dumps(
                [{'idea': idea.title, 'description': idea.description}])
        except:
            response = json.dumps([{'Error': 'No idea with that name'}])
    return HttpResponse(response, content_type="text/json")

@csrf_exempt
def add_idea(request):
    if request.method == 'POST':
        payload = json.loads(request.body)
        title = payload['title']
        description = payload['description']
        idea = Idea(title=title, description=description)
        try:
            idea.save()
            response = json.dumps([{'Success': 'Idea added sucessfylly'}])
        except:
            response = json.dumps([{'Error': 'Idea not added sucessfylly'}])
        return HttpResponse(response, content_type='text/json')
    else:
        response = json.dumps([{}])
        return HttpResponse(response, content_type='text/json')


#!/usr/bin/env python
# --------------------------------------------------------------------
# cli.py
#
# Author: Lain Musgrove (lain.musgrove@hearst.com)
# Date: Monday October 9, 2023
# --------------------------------------------------------------------

import asyncio
import getpass
import shlex
import signal
import ssl
import sys
import threading
import time
import traceback
from argparse import ArgumentParser
from dataclasses import dataclass
from typing import Optional

from bivalve.agent import BivalveAgent
from bivalve.call import Call
from bivalve.logging import LogManager
from bivalve.nio import NonBlockingTextInput

# --------------------------------------------------------------------
log = LogManager().get(__name__)

# --------------------------------------------------------------------
@dataclass
class Config:
    host: Optional[str] = None
    port: Optional[int] = None
    sock: Optional[str] = None
    debug = False
    ssl = False
    verify_cert = True
    check_hostname = True

    def _argparser(self) -> ArgumentParser:
        parser = ArgumentParser(description="bivalve agent client")

        parser.add_argument(
            "--host",
            "-H",
            type=str,
            help="Host to connect to.  Requires --port/-p to be specified.",
        )
        parser.add_argument(
            "--port",
            "-p",
            type=int,
            help="Port on the host to connect to.  Requires --host/-h to be specified.",
        )
        parser.add_argument(
            "--sock", "-s", type=str, help="Path to a UNIX socket file to connect to."
        )
        parser.add_argument(
            "--ssl",
            "-S",
            action="store_true",
            help="Connect to an SSL encrypted socket.",
        )
        parser.add_argument(
            "--debug", action="store_true", help="Print DEBUG level logs."
        )
        parser.add_argument(
            "--no-check-hostname",
            dest="check_hostname",
            action="store_false",
            help="Disable SSL cert hostname check.",
        )
        parser.add_argument(
            "--no-verify-cert",
            dest="verify_cert",
            action="store_false",
            help="Disable SSL cert verification.",
        )

        return parser

    def parse_args(self):
        self._argparser().parse_args(namespace=self)
        if (self.host is None or self.port is None) and (self.sock is None):
            raise ValueError("One of host/port or sock must be specified.")
        return self

# --------------------------------------------------------------------
class ClientAgent(BivalveAgent):
    def __init__(self, config: Config):
        super().__init__()
        self.config = config

    def ctrlc_handler(self, *_):
        log.critical("Ctrl+C received.")
        self.shutdown()

    async def run(self):
        signal.signal(signal.SIGINT, self.ctrlc_handler)

        ssl_ctx: Optional[ssl.SSLContext] = None

        if self.config.ssl:
            ssl_ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
            ssl_ctx.check_hostname = self.config.check_hostname
            ssl_ctx.verify_mode = ssl.CERT_NONE
            if self.config.verify_cert:
                ssl_ctx.verify_mode = ssl.CERT_REQUIRED

        await self.connect(
            host=self.config.host,
            port=self.config.port,
            path=self.config.sock,
            ssl=ssl_ctx,
        )

        loop = asyncio.get_event_loop()
        thread = threading.Thread(target=self.input_thread, args=(loop,))
        thread.start()
        await super().run()
        thread.join()

    def on_disconnect(self, _):
        self.shutdown()

    def on_unrecognized_command(self, conn, *argv):
        print(f"<< CMD {' '.join(argv)}")

    def input_thread(self, loop):
        current_call: Optional[Call] = None

        asyncio.set_event_loop(loop)

        with NonBlockingTextInput() as ninput:
            while not self._shutdown_event.is_set():
                if current_call:
                    try:
                        result = current_call.future.result()

                        if result.code == result.Code.OK:
                            print()
                            print(f"<< OK {' '.join(result.content)}")

                        elif result.code == result.Code.ERROR:
                            print()
                            print(f"<< ERROR {' '.join(result.content)}")

                        current_call = None

                    except asyncio.InvalidStateError:
                        sys.stdout.write(".")
                        sys.stdout.flush()
                        time.sleep(0.25)

                    except Exception:
                        traceback.print_exc()
                        current_call = None
                    continue

                s = ninput.read("> ", timeout=0.25)
                if s is not None:
                    argv_in = shlex.split(s)
                    if not argv_in:
                        continue

                    argv = []

                    while argv_in:
                        arg = argv_in.pop(0)
                        if arg == "<>":
                            argv.append(getpass.getpass("pwd > "))
                        else:
                            argv.append(arg)

                    if argv[0] == "call":
                        current_call = self.call(*argv[1:], timeout_ms=0)

                    else:
                        asyncio.run_coroutine_threadsafe(
                            self.send_to(self.peers(), *argv), loop
                        )

# --------------------------------------------------------------------
def main():
    LogManager().setup()
    config = Config().parse_args()

    if config.debug:
        LogManager().set_level("DEBUG")
    else:
        LogManager().set_level("WARNING")

    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)

    client = ClientAgent(config)
    loop.run_until_complete(client.run())
    print()

# --------------------------------------------------------------------
if __name__ == "__main__":
    main()


import tempfile
import subprocess
import os
import yaml
import codecs
import logging

class FileService(object):
    METAPATH_KEY = "_metapath"

    def __init__(self, logger=None):
        self.logger = logger or logging.getLogger(__name__)

    def edit_temp_file(self, initial_text):
        """Edits a temp file in a subprocess"""
        with tempfile.NamedTemporaryFile(suffix=".tmp") as tfile:
            tfile.write(initial_text)
            tfile.flush()
            self.logger.info("Starting text editor (vim) as a subprocess")
            subprocess.call(["vim", '+set backupcopy=yes', tfile.name])
            self.logger.info("Text editor closed")
            tfile.seek(0)
            ret_bytes = tfile.read().rstrip()
            ret = codecs.decode(ret_bytes, "UTF-8")
            return ret

    def read_line(self, path, line_no):
        current_line = 0
        with open(path, "r") as fs:
            for line in fs:
                current_line += 1
                if current_line == line_no:
                    return line.strip()
        return None

    def get_metapath(self, path):
        metafile_ext = ".yaml"
        directory, fname = os.path.split(path)
        fname_no_ext, ext = os.path.splitext(fname)
        if ext == metafile_ext:
            raise FileServiceException("Can't get a metafile for files using the {} extension"
                                       .format(metafile_ext))
        return os.path.join(directory, fname_no_ext + metafile_ext)

    def get_metafile(self, path):
        """Returns the metafile for this file, or a new one"""
        self.logger.debug("Fetching metafile at {}".format(path))
        metapath = self.get_metapath(path)
        try:
            with open(metapath) as fs:
                metafile = yaml.load(fs)
        except FileNotFoundError:
            metafile = {"path": path, "lines": dict()}

        # Add a path to the object, which should not be serialized
        metafile[self.METAPATH_KEY] = metapath
        return metafile

    def save_metafile(self, metafile):
        self.logger.debug("Saving metafile")
        metapath = metafile[self.METAPATH_KEY]
        with open(metapath, "w") as fs:
            # HACK:
            del metafile[self.METAPATH_KEY]
            yaml.dump(metafile, fs, default_flow_style=False)
            metafile[self.METAPATH_KEY] = metapath

class FileServiceException(Exception):
    """Generic exception involving the file service"""
    pass


#author: Samet Kalkan

import numpy as np
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dropout, Dense
from keras.utils import np_utils
from keras.callbacks import ModelCheckpoint
from keras import regularizers
from keras import backend as K
np.random.seed(0)

train_data = np.load("/home/baker/Desktop/BDD100K/bdd_digitalize/100/7.17/train_data_i.npy")
train_label = np.load("/home/baker/Desktop/BDD100K/bdd_digitalize/100/7.17/train_label_i.npy")

#train_data=train_data[:60000]
#train_label=train_label[:60000]

size = train_data.shape[1]

# normalization
train_data = train_data / 255.0

train_data = train_data.reshape(train_data.shape[0], size, size, 3)

# number of class
num_classes = 7

# for example if label is 4 converts it [0,0,0,0,1]
train_label = np_utils.to_categorical(train_label, num_classes)

val_data=train_data[60000:]
val_label=train_label[60000:]
train_data=train_data[:60000]
train_label=train_label[:60000]

def mycrossentropy(y_true, y_pred, e=0.1):
    return (1-e)*K.categorical_crossentropy(y_true,y_pred) + e*K.categorical_crossentropy(K.ones_like(y_pred)/num_classes, y_pred)

model = Sequential()

#convolutional layer with 5x5 32 filters and with relu activation function
#input_shape: shape of the each data
#kernel_size: size of the filter
#strides: default (1,1)
#activation: activation function such as "relu","sigmoid"

model.add(Conv2D(32, kernel_size=(2,2),input_shape=(size, size, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

for i in range(4):
    model.add(Conv2D(32, kernel_size=(2,2), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=(1,1), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())

# beginning of fully connected neural network.
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.5))
# Add fully connected layer with a softmax activation function
model.add(Dense(num_classes, activation='softmax'))

# Compile neural network
model.compile(loss=mycrossentropy, # self_Cross-entropy
                optimizer='adam', # Root Mean Square Propagation
                metrics=['accuracy']) # Accuracy performance metric

checkpointer = ModelCheckpoint(filepath='modelsCNN/size100/7.17_i_1.h5', verbose=1, save_best_only=True)

# begin train the data
history = model.fit(train_data, # train data
            train_label, # label
            validation_data=[val_data,val_label],
            epochs=40, # Number of epochs
            verbose=1,
            batch_size=256,
            callbacks=[checkpointer]
            )

#model.save("modelsCNN/size100/7.17.h5",overwrite=True)


from collections import deque
import sys
input = sys.stdin.readline
n, m = map(int, input().split())
graph = [[]*(n+1) for _ in range(n+1)]
visited = [False] * (n+1)

for _ in range(m):
    a, b = map(int, input().split())
    graph[a].append(b)
    graph[b].append(a)
cnt = 0

def bfs(v):
    queue = deque([v])
    visited[v] = True
    while queue:
        a = queue.popleft()
        for i in graph[a]:
            if not visited[i]:
                queue.append(i)
                visited[i] = True

for i in range(1, n+1):
    if visited[i] == False:
        bfs(i)
        cnt+=1
print(cnt)

# from collections import deque
# n, m = map(int, input().split())

# graph = [[] for _ in range(n+1)]
# visited = [False] * (n+1)

# for _ in range(m):
#     a, b = map(int, input().split())
#     graph[a].append(b)
#     graph[b].append(a)

# def bfs(start):
#     visited[start] = True
#     queue = deque()
#     queue.append(start)

#     while queue:
#         a = queue.popleft()
#         for i in graph[a]:
#             if not visited[i]:
#                 visited[i] = True
#                 queue.append(i)

# cnt = 0
# for i in range(1, n+1):
#     if not visited[i]:
#         bfs(i)
#         cnt += 1
# print(cnt)


import torch
import tensorflow as tf
import math
import numpy as np
import cv2
from tensorflow import keras

def NME(y_true,y_pred):
    w = 256
    h = 256
    d = (w**2+h**2)**(1/2)

    nose_x = y_true[:,0]
    nose_x_pred = y_pred[:,0]
    nose_y = y_true[:,1]
    nose_y_pred = y_pred[:,1]
    distance_nose = (tf.math.sqrt((nose_x_pred - nose_x)**2 + (nose_y_pred - nose_y)**2))/d
    left_eye_x = y_true[:,2]
    left_eye_x_pred = y_pred[:,2]
    left_eye_y = y_true[:,3]
    left_eye_y_pred = y_pred[:,3]
    distance_left_eye = (tf.math.sqrt((left_eye_x_pred - left_eye_x)**2 + (left_eye_y_pred - left_eye_y)**2))/d
    right_eye_x = y_true[:,4]
    right_eye_x_pred = y_pred[:,4]
    right_eye_y = y_true[:,5]
    right_eye_y_pred = y_pred[:,5]
    distance_right_eye = (tf.math.sqrt((right_eye_x_pred - right_eye_x)**2 + (right_eye_y_pred - right_eye_y)**2))/d
    left_mouse_x = y_true[:,6]
    left_mouse_x_pred = y_pred[:,6]
    left_mouse_y = y_true[:,7]
    left_mouse_y_pred = y_pred[:,7]
    distance_left_mouse = (tf.math.sqrt((left_mouse_x_pred - left_mouse_x)**2 + (left_mouse_y_pred - left_mouse_y)**2))/d
    right_mouse_x = y_true[:,8]
    right_mouse_x_pred = y_pred[:,8]
    right_mouse_y = y_true[:,9]
    right_mouse_y_pred = y_pred[:,9]
    distance_right_mouse = (tf.math.sqrt((right_mouse_x_pred - right_mouse_x)**2 + (right_mouse_y_pred - right_mouse_y)**2))/d
    left_ear_x = y_true[:,10]
    left_ear_x_pred = y_pred[:,10]
    left_ear_y = y_true[:,11]
    left_ear_y_pred = y_pred[:,11]
    distance_left_ear = (tf.math.sqrt((left_ear_x_pred - left_ear_x)**2 + (left_ear_y_pred - left_ear_y)**2))/d
    right_ear_x = y_true[:,12]
    right_ear_x_pred = y_pred[:,12]
    right_ear_y = y_true[:,13]
    right_ear_y_pred = y_pred[:,13]
    distance_right_ear = (tf.math.sqrt((right_ear_x_pred - right_ear_x)**2 + (right_ear_y_pred - right_ear_y)**2))/d

    nme = (tf.reduce_mean(distance_right_ear) + tf.reduce_mean(distance_left_ear) +  tf.reduce_mean(distance_right_mouse) + tf.reduce_mean(distance_left_mouse) + tf.reduce_mean(distance_right_eye) + tf.reduce_mean(distance_left_eye) + tf.reduce_mean(distance_nose))

    return nme

class Image_Crop:
    def __init__(self, yolo_path, model_path):
        self.yolo = torch.load(yolo_path)
        self.model = keras.models.load_model(model_path, custom_objects = {'NME' : NME})

    def predict_landmarks(self, image):

        img_size = 256

        try:
            result = self.yolo(image , augment=True)
            crop_result = result.crop()

            x,y,w,h, = int(crop_result[0]['box'][0]), int(crop_result[0]['box'][1]), int(crop_result[0]['box'][2]), int(crop_result[0]['box'][3])

            crop_img = image[y:y+(h-y), x:x+(w-x)]

            crop_size = crop_img.shape

            # Preprocessing for Prediction
            image = cv2.resize(crop_img, (img_size, img_size))
            image = image.reshape((1,img_size,img_size,3))
            image = image/255.
            predict_landmarks = self.model.predict(image)[0]

            dis_x, dis_y = crop_size[1] ,crop_size[0]
            for i in range(0,14,2):
                cv2.circle(crop_img, (int(predict_landmarks[i]*(dis_x/256)),int(predict_landmarks[i+1] *(dis_y/256))), 5, (0,255,0), 2, cv2.LINE_AA)

            return crop_img

        except:
            return image

    def for_cam(self, image):
        org_img = image
        img_size = 256
        result = self.yolo(image , augment=True)

        try:
            crop_result = result.crop()

            x,y,w,h, = int(crop_result[-1]['box'][0]), int(crop_result[-1]['box'][1]), int(crop_result[-1]['box'][2]), int(crop_result[-1]['box'][3])

            crop_img = image[y:h, x:w]

            crop_size = crop_img.shape

            # Preprocessing for Prediction
            image = cv2.resize(crop_img, (img_size, img_size))
            image = image.reshape((1,img_size,img_size,3))
            image = image/255.
            predict_landmarks = self.model.predict(image)[0]

            for i in range(0,14,2):
                tx = int((predict_landmarks[i])*(crop_size[1]/256) + x)
                ty = int((predict_landmarks[i+1])*(crop_size[0]/256) + y)
                cv2.circle(org_img, (tx,ty), 5, (0,255,0), 2, cv2.LINE_AA)

            return org_img

        except:
            return image


# -*- coding: utf-8 -*-

import tkinter as tk
from typing import List

from ParseError import ParseError
from geometry_syntax import GeometrySyntax
from oval_parameters import OvalParameters
from preview import Preview

class Application(tk.Frame):
    _syntax_error_tag = "syntax_error"

    def __init__(self, master, title: str):
        super().__init__(master)
        self._master = master

        self.grid(sticky="NEWS")
        self.columnconfigure(0, weight=1, uniform="app_column")
        self.columnconfigure(1, weight=1, uniform="app_column")
        self.rowconfigure(0, weight=1)

        self.winfo_toplevel().title(title)
        self._create_widgets()
        self._bind_event_handlers()

    def _create_widgets(self):
        editor_frame = tk.LabelFrame(self, text="Editor")
        editor_frame.grid(row=0, column=0, sticky="NEWS")
        editor_frame.columnconfigure(0, weight=1)
        editor_frame.rowconfigure(0, weight=1)

        preview_frame = tk.LabelFrame(self, text="Preview")
        preview_frame.grid(row=0, column=1, sticky="NEWS")
        preview_frame.columnconfigure(0, weight=1)
        preview_frame.rowconfigure(0, weight=1)

        self.__text_field = tk.Text(
            editor_frame,
            undo=True,
            wrap=tk.WORD,
            font="fixed",
        )
        self.__text_field.grid(sticky="NEWS")
        self.__text_field.tag_config(
            self._syntax_error_tag, background="red", foreground="white"
        )

        self.__preview = Preview(
            preview_frame, self._on_new_figure_created, self._on_figure_updated
        )

    def _bind_event_handlers(self):
        self.__text_field.bind("<Leave>", lambda _: self._on_update_text())
        self.__text_field.bind("<FocusOut>", lambda _: self._on_update_text())
        self.__text_field.bind("<Key-Return>", lambda _: self._on_update_text())

    def _on_update_text(self):
        self._remove_syntax_errors()
        text = self.__text_field.get("1.0", tk.END)
        figures, errs = GeometrySyntax.parse(text)
        failed_line_no = self.__preview.draw_figures(figures)
        errs += list(map(lambda l: ParseError(line=l), failed_line_no))
        self._display_errors(errs)

    def _display_errors(self, errors: List[ParseError]):
        self._remove_syntax_errors()
        for err in errors:
            self.__text_field.tag_add(
                self._syntax_error_tag,
                f"{err.line}.0",
                f"{err.line}.0 + 1l - 1c",
            )

    def _remove_syntax_errors(self):
        x = self.__text_field.tag_ranges(self._syntax_error_tag)
        for start, end in zip(x[::2], x[1::2]):
            self.__text_field.tag_remove(self._syntax_error_tag, start, end)

    def _on_new_figure_created(self, params: OvalParameters):
        new_text = GeometrySyntax.serialize([params])
        line_count = self._count_lines()
        if (
            len(self.__text_field.get(f"{line_count}.0", f"{line_count}.0 + 1l - 1c"))
            > 0
        ):
            new_text = "\n" + new_text
        self.__text_field.insert(tk.END, new_text)
        self._on_update_text()

    def _on_figure_updated(self, params: OvalParameters):
        new_text = GeometrySyntax.serialize([params])
        edited_line = params.id
        line_count = self._count_lines()
        if line_count >= edited_line:
            start = f"{edited_line}.0"
            end = f"{edited_line}.0 + 1l - 1c"
            self.__text_field.delete(start, end)
            self.__text_field.insert(start, new_text)
            self._on_update_text()

    def _count_lines(self) -> int:
        return int(self.__text_field.index(tk.END + "-1c").split(".")[0])

    def _on_exit(self):
        self._master.destroy()


from __future__ import absolute_import
from __future__ import print_function

import pyspark
import h5py
import json
from keras.optimizers import serialize as serialize_optimizer
from keras.models import load_model

from .utils import lp_to_simple_rdd
from .utils import model_to_dict
from .mllib import to_matrix, from_matrix, to_vector, from_vector
from .optimizers import SGD
from .worker import AsynchronousSparkWorker, SparkWorker
from .parameter import HttpServer, SocketServer
from .parameter import HttpClient, SocketClient

class SparkModel(object):

    def __init__(self, model, mode='asynchronous', frequency='epoch',  parameter_server_mode='http', num_workers=None,
                 elephas_optimizer=None, custom_objects=None, batch_size=32, *args, **kwargs):
        """SparkModel

        Base class for distributed training on RDDs. Spark model takes a Keras
        model as master network, an optimization scheme, a parallelisation mode
        and an averaging frequency.

        :param model: Compiled Keras model
        :param mode: String, choose from `asynchronous`, `synchronous` and `hogwild`
        :param frequency: String, either `epoch` or `batch`
        :param parameter_server_mode: String, either `http` or `socket`
        :param num_workers: int, number of workers used for training (defaults to None)
        :param elephas_optimizer: Elephas optimizer
        :param custom_objects: Keras custom objects
        """

        self._master_network = model
        if not hasattr(model, "loss"):
            raise Exception(
                "Compile your Keras model before initializing an Elephas model with it")
        metrics = model.metrics
        loss = model.loss
        optimizer = serialize_optimizer(model.optimizer)

        if custom_objects is None:
            custom_objects = {}
        if metrics is None:
            metrics = ["accuracy"]
        if elephas_optimizer is None:
            self.optimizer = SGD()
        else:
            self.optimizer = elephas_optimizer
        self.mode = mode
        self.frequency = frequency
        self.num_workers = num_workers
        self.weights = self._master_network.get_weights()
        self.pickled_weights = None
        self.master_optimizer = optimizer
        self.master_loss = loss
        self.master_metrics = metrics
        self.custom_objects = custom_objects
        self.parameter_server_mode = parameter_server_mode
        self.batch_size = batch_size
        self.kwargs = kwargs

        self.serialized_model = model_to_dict(self.master_network)
        # TODO only set this for async/hogwild mode
        if self.parameter_server_mode == 'http':
            self.parameter_server = HttpServer(
                self.serialized_model, self.optimizer, self.mode)
            self.client = HttpClient()
        elif self.parameter_server_mode == 'socket':
            self.parameter_server = SocketServer(self.serialized_model)
            self.client = SocketClient()
        else:
            raise ValueError("Parameter server mode has to be either `http` or `socket`, "
                             "got {}".format(self.parameter_server_mode))

    @staticmethod
    def get_train_config(epochs, batch_size, verbose, validation_split):
        return {'epochs': epochs,
                'batch_size': batch_size,
                'verbose': verbose,
                'validation_split': validation_split}

    def get_config(self):
        base_config = {
            'parameter_server_mode': self.parameter_server_mode,
            'elephas_optimizer': self.optimizer.get_config(),
            'mode': self.mode,
            'frequency': self.frequency,
            'num_workers': self.num_workers,
            'batch_size': self.batch_size}
        config = base_config.copy()
        config.update(self.kwargs)
        return config

    def save(self, file_name):
        model = self.master_network
        model.save(file_name)
        f = h5py.File(file_name, mode='a')

        f.attrs['distributed_config'] = json.dumps({
            'class_name': self.__class__.__name__,
            'config': self.get_config()
        }).encode('utf8')

        f.flush()
        f.close()

    @property
    def master_network(self):
        return self._master_network

    @master_network.setter
    def master_network(self, network):
        self._master_network = network

    def start_server(self):
        self.parameter_server.start()

    def stop_server(self):
        self.parameter_server.stop()

    def predict(self, data):
        """Get prediction probabilities for a numpy array of features
        """
        return self.master_network.predict(data)

    def predict_classes(self, data):
        """ Predict classes for a numpy array of features
        """
        return self.master_network.predict_classes(data)

    def fit(self, rdd, epochs=10, batch_size=32,
            verbose=0, validation_split=0.1):
        """
        Train an elephas model on an RDD. The Keras model configuration as specified
        in the elephas model is sent to Spark workers, abd each worker will be trained
        on their data partition.

        :param rdd: RDD with features and labels
        :param epochs: number of epochs used for training
        :param batch_size: batch size used for training
        :param verbose: logging verbosity level (0, 1 or 2)
        :param validation_split: percentage of data set aside for validation
        """
        if self.num_workers:
            rdd = rdd.repartition(self.num_workers)

        if self.mode in ['asynchronous', 'synchronous', 'hogwild']:
            self._fit(rdd, epochs, batch_size, verbose, validation_split)
        else:
            raise ValueError(
                "Choose from one of the modes: asynchronous, synchronous or hogwild")

    def _fit(self, rdd, epochs, batch_size, verbose, validation_split):
        """Protected train method to make wrapping of modes easier
        """
        self.master_network.compile(optimizer=self.master_optimizer,
                                    loss=self.master_loss,
                                    metrics=self.master_metrics)
        if self.mode in ['asynchronous', 'hogwild']:
            self.start_server()
        train_config = self.get_train_config(
            epochs, batch_size, verbose, validation_split)
        mode = self.parameter_server_mode
        freq = self.frequency
        optimizer = self.master_optimizer
        loss = self.master_loss
        metrics = self.master_metrics
        custom = self.custom_objects

        yaml = self.master_network.to_yaml()
        init = self.master_network.get_weights()
        parameters = rdd.context.broadcast(init)

        if self.mode in ['asynchronous', 'hogwild']:
            worker = AsynchronousSparkWorker(
                yaml, parameters, mode, train_config, freq, optimizer, loss, metrics, custom)
            rdd.mapPartitions(worker.train).collect()
            new_parameters = self.client.get_parameters()
        elif self.mode == 'synchronous':

            worker = SparkWorker(yaml, parameters, train_config,
                                 optimizer, loss, metrics, custom)
            deltas = rdd.mapPartitions(worker.train).collect()
            new_parameters = self.master_network.get_weights()
            for delta in deltas:
                def base_constraint(a): return a
                constraints = [base_constraint for _ in self.weights]
                new_parameters = self.optimizer.get_updates(
                    self.weights, constraints, delta)
        else:
            raise ValueError("Unsupported mode {}".format(self.mode))
        self.master_network.set_weights(new_parameters)
        if self.mode in ['asynchronous', 'hogwild']:
            self.stop_server()

def load_spark_model(file_name):
    model = load_model(file_name)
    f = h5py.File(file_name, mode='r')

    elephas_conf = json.loads(f.attrs.get('distributed_config'))
    class_name = elephas_conf.get('class_name')
    config = elephas_conf.get('config')
    if class_name == "SparkModel":
        return SparkModel(model=model, **config)
    elif class_name == "SparkMLlibModel":
        return SparkMLlibModel(model=model, **config)

class SparkMLlibModel(SparkModel):

    def __init__(self, model, mode='asynchronous', frequency='epoch', parameter_server_mode='http',
                 num_workers=4, elephas_optimizer=None, custom_objects=None, batch_size=32, *args, **kwargs):
        """SparkMLlibModel

        The Spark MLlib model takes RDDs of LabeledPoints for training.

        :param model: Compiled Keras model
        :param mode: String, choose from `asynchronous`, `synchronous` and `hogwild`
        :param frequency: String, either `epoch` or `batch`
        :param parameter_server_mode: String, either `http` or `socket`
        :param num_workers: int, number of workers used for training (defaults to None)
        :param elephas_optimizer: Elephas optimizer
        :param custom_objects: Keras custom objects
        """
        SparkModel.__init__(self, model=model, mode=mode, frequency=frequency,
                            parameter_server_mode=parameter_server_mode, num_workers=num_workers,
                            elephas_optimizer=elephas_optimizer, custom_objects=custom_objects,
                            batch_size=batch_size, *args, **kwargs)

    def fit(self, labeled_points, epochs=10, batch_size=32, verbose=0, validation_split=0.1,
            categorical=False, nb_classes=None):
        """Train an elephas model on an RDD of LabeledPoints
        """
        rdd = lp_to_simple_rdd(labeled_points, categorical, nb_classes)
        rdd = rdd.repartition(self.num_workers)
        self._fit(rdd=rdd, epochs=epochs, batch_size=batch_size,
                  verbose=verbose, validation_split=validation_split)

    def predict(self, mllib_data):
        """Predict probabilities for an RDD of features
        """
        if isinstance(mllib_data, pyspark.mllib.linalg.Matrix):
            return to_matrix(self.master_network.predict(from_matrix(mllib_data)))
        elif isinstance(mllib_data, pyspark.mllib.linalg.Vector):
            return to_vector(self.master_network.predict(from_vector(mllib_data)))
        else:
            raise ValueError(
                'Provide either an MLLib matrix or vector, got {}'.format(mllib_data.__name__))


from matplotlib import pyplot as plt
from PIL import Image

img = Image.open("original.jpg")

img2 = Image.open("broken1.png").convert(img.mode)

img2 = img2.resize(img.size)

img3 = Image.blend(img,img2,0.35)

plt.figure(num='BROKEN LENS Failure')
plt.subplot(121),plt.imshow(img),plt.title('Original')
plt.xticks([]), plt.yticks([])
plt.subplot(122),plt.imshow(img3),plt.title('Broken Lens')
plt.xticks([]), plt.yticks([])
plt.show()